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Pre4  ace 


The  purpose  o-f  this  study  was  to  investigate  the  effect 
that  different  aggregation  methods  have  on  the  group  priority 
vector  determined  by  the  Analytical  Hierarchy  Process  (AHP). 
Three  aggregation  methods  were  applied  to  a  previously 
accomplished  study  to  determine  the  rank  order  correlation 
for  the  respective  priority  vectors  and  consistency  measures. 

The  previous  study  was  the  Battel le  Columbus 
Laboratories’  1993  Trans  Atmospheric  Vehicle  (TAV) 
feasibility  analysis  for  the  Aeronautical  Systems  Division  of 
USAF  Systems  Command.  The  "revised"  Battel le  study  provided  a 
hierarchy  of  32  criteria  to  evaluate  14  TAV  design  proposals. 
The  judges  were  11  of  the  original  13  TAV  steering  committee 
members  who  responded  to  Battel le’s  questionnaire  to  provide 
the  top  level  objective  weights. 

The  first  aggregation  technique  determined  the  geometric 
mean  of  the  12  input  comparisons  prior  to  the  calculation  of 
eigenvector  (priority  vector).  The  second  method  determined 
the  arithmetic  mean  of  the  judge’s  priority  vectors.  The 
third  method  converted  the  AHP  input  ratios  to  ordinal 
preferences  and  determined  the  group  priority  vector  by  the 
percentage  of  "votes"  for  each  abjective.  The  examination  of 
the  geometric  mean  of  the  priority  vectors  was  not  examined 


in  this  research 


The  results  showed  that  the  three  aggregation  techniques 


produced  correlated  priority  vectors.  Also,  the  consistency 
measures  for  the  ordinal  matrices  and  the  AHP  matrices  were 
highly  correlated.  Qraphical  analysis  indicated  that  ratio 
"distance'*  may  be  preserved  between  preferences  when  ratio 
scale  inputs  are  converted  to  ordinal  preferences.  However, 
this  interpretation  must  consider  the  bias  caused  by  creating 
the  "voting"  matrix  directly  from  the  AHP  inputs,  rather  than 
rather  than  from  an  independent  survey.  Further  study  will  be 
necessary  to  determine  the  number  of  judges  and  the  number  of 
criteria  necessary  for  this  preservation  of  distance. 

In  performing  the  surveys  and  writing  this  thesis  I  have 
had  a  great  deal  of  help  from  others.  I  am  indebted  to  my 
faculty  advisor,  LtCol  Mark  Mekaru,  and  my  reader.  Major  Ken 
Feldman  for  their  insight  and  lifting  spirits.  I  also  wish  to 
thank  members  of  the  TAV  steering  committee  for  their 
expeditious  assistance  in  this  endeavor,  with  particular 
thanks  to  Dr.  Eric  Rice  and  Dr.  Jerry  Arnett  for  their 
invaluable  information  and  advice.  Finally,  1  wish  to  thank 
my  wife  Gerry  for  maintaining  an  even  keel  during  these 
stormy  si x  monthes. 
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Abstract 

This  study  showed  that  Analytical  Hierarchy  Process 
(AHP)  group  priority  vector  was  independent  a-f  the  three 
aggregation  techniques  examined.  Also,  the  (AHP)  consistency 
index  was  correlated  with  Kendall’s  Coefficient  of 
Consistence.  The  study  applied  the  AHP  to  Battel le  Columbus 
Laboratories  1983  TAV  study  using  11  of  the  original  members 
of  the  1983  TAV  steering  committee. 

The  first  aggregation  technique  used  the  geometric  mean 
of  the  12  input  comparisons  to  calculate  the  group  priority 
vector.  The  second  method  used  the  arithmetic  mean  of  the 
judges’  priority  vectors  to  calculate  the  group  priority 
vector.  The  third  method  converted  the  AHP  input  ratios  to 
ordinal  preferences  and  used  the  percentage  of  preferences 
for  each  objective  to  determine  the  group  priority  vector. 

The  analysis  used  Apple  Pascal  computer  code  for 
manipulating  the  AHP  input  files.  Kendall’s  correlation 
coefficient,  tau,  showed  significant  rank  order  correlation 
between  the  three  group  vectors.  Also,  graphical  analysis 
indicated  that  strength  of  preference  was  preserved  when 
converting  from  a  ratio  to  ordinal  scale.  Minor 
dissatisfaction  with  the  priority  vectors  suggests  that  when 
administering  the  AHP  bv  survey,  the  comparison  scale  should 
be  displayed  to  reflect  the  logarithmic  nature  of  the  input 


di-fferant  aggregation  methods.  The  examination  of  the 
statistical  properties  of  the  Kemeny  and  Snell  correlation  is 
outside  the  realm  of  this  study.  However ,  the  Kemeny  and 
Snell  correlation’s  similarity  to  tau  supports  the  use  of  tau 
as  a  statistical  evaluation  in  this  thesis.  The  discussion  on 
tau  as  a  correlation  measure  between  two  rank  orders  will  be 
in  chapter  IV,  Methodology. 

Ogini gQ_El.i.ci tati.gns.  Robert  Fallon  showed  in 
"Subjective  Assessment  of  Uncertainty"  that  biases  in  a 
person’s  background  affect  his  prediction  of  future  events. 

He  showed  how  current  availability,  anchoring,  and 
representativeness  distort  one’s  assessment  of  uncertainty  in 
probabilistic  terms  (Quade,  1970:287).  Although  there  is 
little  one  can  do  to  diminish  this  bias,  the  analyst  must  be 
aware  that  despite  perfect  consensus  among  undisputed 
experts,  there  are  inherent  biases  that  may  distort  the 
entire  prediction.  An  example  would  be  the  error  in 
predicting  the  cost  of  1982  petroleum  during  the  1973  oil 
embargo. 

Saakman  criticizes  the  entire  process  of  eliciting 
expert  opinions.  Directed  at  the  Delphi  technique,  the 
criticisms  apply  equally  well  to  any  of  the  elicitation 
methods  discussed  in  this  thesis.  Saakman  points  out  the 
following  weaknesses  in  soliciting  opinions  from  experts: 

The  questionnaires  should  be  administered  under 

rigorously  controlled  conditions. 
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chapter 


SiliSQSC-EUQC^igQSi  in  1962  Keiiieny  and  Snell  began  a 
reeearch  thrust  in  priority  vectors  achieved  by  ranking.  They 
proposed  that  comnittee  rankings  be  vieead  in  terms  of  a 
"distance"  measure.  Such  a  measure  relative  to  a  pair  of 
rankings  would  be  an  indicator  of  the  degree  of  correlation 
between  rankings.  They  established  a  set  of  axioms  which  any 
such  measure  should  satisfy,  and  proved  its  existence  and 
uniqueness.  Their  set  of  axioms  is  similar  to  Arrow’s  axioms, 
except  the  former  does  not  require  the  "irrelevant 
alternatives"  condition  (Kemeny  and  Snell,  1962). 

Kemeny  and  Snell  used  their  distance  function  to  propose 
the  median  and  mean  rankings  as  acceptable  forms  of 
consensus.  Bogart,  however,  showed  that  in  the  general 
setting  Kemeny  and  Snell’s  means  and  medians  were  not 
necessarily  unique.  It  would  be  necessary  to  restrict  the 
orderings  to  an  odd  number  to  achieve  uniqueness  (Bogart, 
1973:65) . 

Bogart  showed  that  in  the  case  of  linear  orderings,  the 
Kemeny  and  Snell  correlation  measure  was  the  same  as 
Kendall’s  tau  (Bogart  1973:44).  However,  for  partial 
orderings  (indifference)  its  value  diverges  from  tau.  Bogart 
suggests  further  study  of  the  properties  of  this  statistic  in 
the  cases  in  which  it  differs  from  tau. 

This  research  does  not  address  Bogart’s  suggested  study 
because  the  focus  is  on  comparing  priority  vectors  derived  by 
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dacisican  makar,  be  she  the  group  itself  through  an 
interactive  process,  or  a  benevolent  dictator,  to  weight  the 
inputs  of  the  individuals.  That  is,  the  supra  nay  give  twice 
the  weight  to  Joe's  opinions  over  Fred's  opinions  because  Joe 
has  30  rare  years  of  experience.  This  process  to  conbine 
individual  values  into  a  group  value  is  more  fully  discussed 
under  Value  Theory,  this  chapter.  This  research  will  attempt 
to  apply  interpersonal  value  comparisons  to  achieve  "more 
rational"  (transitive)  group  decisions,  as  explained  in 
Chapter  IV,  Methodology. 

Arrow  showed  that  group  decisions  may  be  intransitive 
and  Keeney  showed  that  the  intransiti vi ty  may  be  reduced  by 
interpersonal  value  comparisons.  The  critical  issue  is  the 
degree  of  intransitivity  of  the  group  decision.  The  more 
intransitive  the  group  decision  is,  the  less  it  reflects  the 
individual  opinions  which  leads  to  a  lack  of  confidence  in 
the  results. 

This  thesis  will  apply  interpersonal  value  comparisons 
to  reduce  the  intransitivity  of  the  group  decision.  In 
addition,  this  thesis  will  apply  two  aggregation  techniques 
that  measure  the  intransi ti vi ty  in  a  series  of  pairwise 
comparisons.  Saaty's  Analytical  Hierarchic  Process  (AHP) 
measures  the  consistency  in  the  strength  of  preferences  by 
the  consistency  index  (Cl).  Kendall's  coefficient  of 
consistence,  zeta,  measures  the  intransitivity  of  voting 
matrices.  Both  of  these  methods  are  explored  in  the  next 
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inpossibla.  ArroM'a  ccxiditi ana  ara  aunmarizod  by  Blint 


1)  Any  anal ganat ion  nathod  auat  dafina  a  uniqua 
ardar  (rationality). 

2)  If  ana  altarnativa  riaaa  or  renaina  stationary 
in  ordor  of  ovary  individual,  than  it  must  not  fall 
in  tho  joint  ordor  (poaitiva  association). 

3)  Tha  addition  or  ronoval  of  an  altarnativa,  which 
roBults  in  no  changa  in  any  individual  ordar  of  the 
ranaining  al ternati vos,  must  not  causa  a  changa  in 
the  ordor  of  tha  remaining  alternatives  of  the 
joint  order  (independence  of  irrelevant 

al ternati ves) . 

4)  The  joint  order  is  a  function  of  the  individual 
orders  and  must  not  be  imposed  by  some  outside 
influence  (citizen’s  sovereignty,  or  pareto 
optimality) . 

5)  The  joint  order  must  not  be  arbitrarily  defined 
by  the  order  of  one  individual  without 
consideration  of  the  other  individual  orders 
(non-dictatorship). 

(Blin  and  Satterthwai te,  1978:252) 


Arrow  proved  that  there  is  no  rule  for  combining  the 
individual’s  rankings  that  is  consistent  with  the  seemingly 
innocuous  conditions.  That  is,  there  is  no  procedure  that  can 
combine  several  individuals’  rankings  of  alternatives  that 
will  satisfy  the  five  conditions  above. 

Arrow’s  Impossibility  Theorem  implies  there  is  no 
procedure  for  combining  individual  rankings  into  a  personal 
comparison  of  preferences  (Keeney  and  Raiffa,  1976:524). 
Keeney,  however,  reports  that  group  decisions  can  be 
transitive  if  there  is  an  additional  condition:  interpersonal 
value  comparisons  are  allowed  (Keeney  and  Raiffa,  1976:145). 

Interpersonal  value  comparisons  allow  the  "supra" 


Ch>ptT  1 1 .  .  4*  t«r*turg  R«vi  aw 


iQtrs^Cilgn 

A  survay  of  rscsnt  litaratura  rsvaals  nunerous  msthods 
for  soliciting  individual  opinion  and  aggregating  it  into  a 
group  consensus.  This  review  will  divide  the  studies  in  three 
broad  categories:  1)  General:  includes  Arrow’s  i mpossi bl i ty 
theorcn  and  the  nechanics  of  elicitation  -  rankings,  ratings, 
and  pairwise  conparisons,  2)  Value  Theory:  includes  an 
explanation  of  the  AHP  and  Tuzxy  set  theory,  and  3)  Group 
aspects:  includes  dynamics  of  groups,  individual  emphasis, 
and  an  explanation  of  Delphi  and  variations. 

0CC6M&  Kenneth  Arrow  first  axiomized  majority  rule 
group  decisions  in  19S1  (Arrow,  1963:12).  He  proved  that  the 
aggregation  of  rational  judges  results  in  an  irrational  group 
decision.  Arrow’s  "Impossibility  Theorem"  showed  that  when 
the  rational  conditions  of  economic  social  choice  were 
defined,  they  gave  inconsistent  group  priority  rankings.  That 
is,  the  aggregate  choice  of  transitive  judges  would  be 
intransitive  (Blin  and  Satterthwai te,  1978:260). 

Many  theoretical  writers  responded  to  Arrow’s  n  nditions 
first  published  in  19S1.  Responding  to  their  criticisms. 

Arrow  strengthened  three  of  the  five  conditions  (1,2,  and  5), 
but  still  showed  that  a  rational  group  decision  was 
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th«  litaratura  raviaw  o#  partinant  aurticlaa  and  taxts  that 
rafaranca  tha  topics  undar  avaluation,  with  particular 
aaphaais  on  Dal phi,  Valua  Thaory,  and  tha  AHP.  Chaptar  Thraa 
ralataa  tha  aspacta  o^f  soliciting  and  aggragating  opinions  in 
the  AHP •  Chapter  Four  di scussas  the  methodol ogy  used  i n  thi s 
thesis  and  chaptar  Six  interprets  the  results  and  dicussas 
the  insights  gained,  along  with  a  few  -folloM  on  research 
ideas. 


FIGURE  2 
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priority  voctora  and  conaiatancy  aaaauraa.  Ralatad  aida 
iaauaa  Mill  ba  axplorad*  auch  aa  diacovaring  tha  affact  that 
applying  Oalphi  haa  on  tha  AHP  Maighta  and  datarnining  tha 
ralationahip  bataaan  Saaty’a  aaaaura  of  ratio  conaiatancy  and 
Kandall’s  naaaura  of  ordinal  conaiatancy  (tranaiti vity) . 

Tha  franaMork  uaad  in  thia  atudy  aaa  tha  Battalia 
hiararchy.  Tha  nunbar  of  criteria  in  the  Battalia  heirarchy 
waa  reduced  fron  47  to  32  aa  a  reault  of  inaight  gained  from 
the  Battalia  atudy  and  to  elininate  redundant  criteria.  Tha 
folloMing  elanenta  Mare  changed  in  the  “reviaed  hierarchy," 
aa  aeen  in  Figure  2: 


Boala  and  Requirenente  Mere  combined  becauae  of  the 
similar  attributes  and  Meighting. 

Environmental  Impact  was  deleted  because  of  the  low 
wei ghting. 

Manned  Flight  was  dropped  because  it  failed  to 
distinguish  between  any  alternative  (All  TAV  designs  were 
capable) . 

5  Minute  Launch  and  2  Missions  per  Day  were  dropped 
because  it  failed  to  distinguish  between  any  alternative  (All 
TAV  designs  were  capable). 

Global  Range  and  Number  of  Expendables  were  dropped 
because  its  attributes  were  the  same  as  single  stage. 

Few  Orbits  was  dropped  because  its  attributes  were 
redundant  with  Loiter  capability. 

Multiple  Dispersal  Sites  was  dropped  because  its 
attributes  were  redundant  with  Mobile  Launch  (runway  length 
required) . 


Overview 

This  thesis  is  divided  into  six  chaipters.  Chapter  Two  is 
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4mM  studiM  on  hoN  to  aggrogato  multi  pi  ■  dmcision  makers 
using  thm  AHP.  Several  questions  surface,  such  ass  Hom 
effective  are  aggregation  methods  that  use  feedback,  such  as 
Delphi,  when  they  are  applied  with  the  AHP?  How  does  the 
priority  vector  change  if  the  mean  is  taken  of  the  pairwise 
comparisons,  rather  than  the  nean  of  the  priority  vectors? 

How  do  these  "before"  and  “after"  aggregated  AHP  vectors 
compare  with  established  aggregation  techniques,  such  as 
majority  rule? 

A  related  question  is  to  determine  the  relationship 
between  the  different  measures  of  consistency.  The  ordinal 

consistency,  that  is  the  degree  of  intransi ti vi ty,  of  voting 

matrices  may  be  measured  by  Kendall's  coefficient  of 

consistence  (zeta) .  Zeta  is  a  count  of  the  number  of  circular 

triads.  The  interval  or  ratio  consistency  is  measured  by  the 

AHP  consistency  indeM  (Cl).  The  Cl  measures  the  divergence  of 

the  individual  comparison  ratios  from  the  corresponding 

priority  vector  ratios.  This  study  will  examine  the 

relationship  between  zeta  and  Cl;  their  correlation  as  a 

function  of  matrix  size,  judge  identifier,  and  varianca 

between  judges. 

RfSfgrch_Qbj.ec£iyes 

This  thesis  will  make  a  critical  examination  of  the  AHP 
applied  to  group  decision  making  using  the  1983  Batelle  study 
of  the  TAV  feasibility.  The  main  objective  is  to  measure  the 
effect  that  different  aggregation  techniques  have  on  the 
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is  liksly  tha  dscision  naksrs  Mould  msks  dif-farant  ratings  i-f 
thay  knaM  tha  attributas  that  Battalia  dafinad  for  tha  study. 
For  SMaopla,  Battalia  datarminad  that  "runMay  langth 
raquirad"  would  ba  tha  driving  attributa  for  "survivability," 
but  tha  author  doubts  that  tha  axparts  usad  this  attributa 
whan  making  thair  assassaant  of  "survivability.” 

Tha  source  of  the  variance  batwaen  objective  ratings  is 
obscured  as  a  result  of  the  judges  not  using  the  same 
attributas.  That  is,  it  is  difficult  to  determine  if  the 
variance  between  the  judges*  ratings  cama  "artificially"  from 
different  assumed  attributes,  or  "honestly”  from  different 
evaluations  of  the  same  attributes.  The  resulting  priority 
vectors  cannot  be  summed  and  averaged  bexzause  the  judges  may 
not  have  used  the  same  attribute  definition. 

□ne  method  to  insure  the  judges  are  using  the  same 
attributes  is  to  ask  them  to  evaluate  the  lower  objectives 
prior  to  comparing  the  next  higher  level  objectives.  This 
will  insure  that  each  judge  is  aware  of  the  subob jecti ves 
that  make  up  the  next  higher  level  abjective.  The  analytical 
hierarchy  process,  applied  from  the  bottom  level  up, 
accomplishes  this  goal.  Each  judge  is  asked  to  compare 
clusters  of  objectives  regarding  their  importance  to  the  next 
level  abjective. 

Re sear ch_ Pr  gb iem 

Despite  the  fact  that  the  AHP  is  gaining  widespread 
acceptance  as  a  method  to  solicit  priorities,  there  has  been 
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Battvlla  usmJ  aqual  weights  -for  all  but  three  criteria! 
propellant  logistics,  and  structural  risk,  and  payload 
performance.  These  criteria  ware  weighted  by  a  factor  of  two. 

Battel le  sent  their  questionnaire  to  the  18  neeber  TAV 
steering  committee  to  elicit  their  ratings  of  the  eight  top 
level  objectives  (on  a  scale  from  1  to  10).  They  received  13 
responses  as  shown  in  Table  1.  The  poll  was  administered  once 
and  the  arithmetic  mean  for  each  objective  rating  was 
normalized  to  obtain  the  weighting  factors.  The  judges  gave 
their  ratings  before  Battel le  had  determined  the  attributes 
to  measure  these  objectives. 

TABLE  1 

Top  Level  Complete  Battel le  Ratings 


Judge 


□bj 

1 

2 

3 

4 

6 

7 

8 

9 

10 

11 

12 

13 

14 

Ave 

1 

10 

10 

10 

10 

10 

10 

10 

7 

10 

10 

10 

10 

10 

9.7 

2 

6 

8 

7 

6 

6 

9 

5 

5 

10 

7 

8 

7 

9 

7.2 

3 

6 

8 

8 

10 

8 

8 

10 

9 

6 

8 

7 

7 

10 

8.  1 

4 

8 

9 

10 

10 

7 

S 

8 

9 

4 

6 

lO 

8 

7 

7.8 

5 

6 

5 

10 

7 

5 

7 

6 

9 

6 

6 

8 

8 

8 

7.0 

6 

8 

5 

6 

7 

7 

6 

10 

8 

8 

7 

9 

8 

9 

7.5 

7 

8 

5 

9 

8 

8 

10 

8 

7 

7 

6 

4 

9 

8 

7.5 

8 

5 

1 

5 

5 

2 

5 

S 

4 

3 

2  3  5 

(Battel le. 

3 

1983: 

3.7 

6-2) 

Note: 

Judge  #13  and  14  not  used  for  subsequent  surveys. 


The  problem  with  this  approach  was  that  the  judges  did 
not  know  the  ranges  of  attributes  that  made  up  the  criteria 
(Keeney,  1977:286).  Decision  makers  must  not  only  know  the 
ranges  of  the  attributes,  but  also  they  must  use  the  same 
attributes  before  comparable  assessments  can  be  elicited.  It 
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critaria  from  thair  in-houu  «Mp«rts.  As 


fan  in  Figura 


Figure  1 
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determine  the  expert’s  priority  vector.  Despite  the  strengths 
o4  the  AHP,  it  is  a  di-fficult  tool  to  aggregate  judges’ 
assessments  because  of  the  enormous  effort  required  to 
achieve  a  group  consensus  of  the  numerous  comparisons.  This 
effort  is  even  more  cumbersome  when  applied  by  mailed  survey, 
as  in  this  study. 

This  study  Mill  examine  different  techniques  to 

aggregate  the  preferences  derived  from  the  AHP,  including 
versions  of  Delphi  to  reduce  spurious  variance  between 
judges,  self  expertise  evaluations  for  individual  weightings, 
and  various  mathmatical  aggregation  methods  to  derive  the 
'‘means."  The  vehicle  used  to  compare  these  different  methods 
will  be  the  1983  Battel le  study. 

§f|tie].le_Stu{ty 

Battel le  sought  to  determine  the  feasibility  of  the 
TAV  concept  and  to  identify  which  designs  were  worth 
pursuing.  They  evaluated  the  effectiveness  of  14  TAV  designs 
using  the  following  weighted  objectives:  Requirements,  Goals, 
Performance,  Costs,  Technological  Risks,  Survivability, 
Logistics,  and  Environmental  Impact. 

These  “top  level"  objectives  were  broken  into  49 
criteria  as  seen  in  Figure  1.  The  entire  hierarchy  was 
established  by  Battel le  except  the  criteria  that  made  up 
requirements  and  goals.  The  requirements  and  goals  criteria 
were  established  by  Mission  Area  Analysis  at  Wright  Patterson 
AFB  (ASD/XRS).  Battelle  determined  the  attributes  for  these 


Various  authors  have  studisd  tha  probl an  of  combining 
individual  prafarancas  into  a  group  choica  or  consansus. 
Problams  of  this  natura  arisa  fraquantly  in  a  variaty  of 
araasi  markating  of  naM  products,  allocation  of  prioritias  to 
R  &  0  projacts,  ate.  In  most  of  thasa  casas,  an  individual’s 
prafaranca  is  axprassad  in  tarns  of  ranking  availabla 
altarnati ves.  The  objactiva  in  each  case  is  to  combine  the 
individual  preferences  into  a  group  consansus  (Cook  and 
Seiford,  1978:1721). 

There  are  two  fundiimental  problems  with  gathering  group 
opinions:  1)  How  does  one  elicit  the  true  individual  opinion? 
and  2)  How  does  one  aggregate  them  to  accurately  represent 
the  group?  Every  analysis  which  uses  expert  opinions  must 
confront  and  answer  these  problems. 

In  1983  Battel le  Corporation  confronted  these  problems 
in  a  feasibility  study  of  the  Trans  -  Atmospheric  Vehicle 
(TAV)  for  the  Aeronautical  Systems  Division  (ASD)  of  the 
United  States  Air  Force.  They  elicited  expert  ratings  to 
establish  the  weights  for  the  top  level  criteria.  They  usked 
each  respondent  to  rate  each  criterion  on  a  scale  of  1  to  10. 
Battel le  aggregated  the  ratings  by  taking  the  arithmetic 
average  for  each  criterion. 

This  thesis  will  explore  the  methods  used  to  solicit  and 
aggregate  expert  opinions.  The  focus  will  be  on  the 
Analytical  Hierarchy  Process  (AHP)  to  solicit  expert  weights. 
The  AHP  uses  pairwise  comparisons  between  the  objectives  to 
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AGGREGATING  INDIVIDUAL  PREFERENCES  IN  THE 
ANALYTIC  HIERARCHY  PROCESS 
APPLIED  TO  THE  1983  BATTELLE  TAV  STUDY 


ChaBt.er_l|._BaclcgroiJijd 


Introduction 

Expert  judgment  must  often  be  used  to  choose  among 
several  alternate  courses  of  action.  An  expert’s  presumed 
intuitive  insight  provides  answers  to  complex  questions  that 
today’s  quantitative  approaches  are  often  unable  to  answer. 

There  are  several  methods  to  elicit  the  expert’s 
opionions.  Ranking  and  rating  are  often  used  in  situations 
which  do  not  allow  a  structured  decision  approach.  Such 
problems  are  usually  characterized  by  lack  of  measurable 
objectives  and  reliable  information  (Baas,  1977:47).  The 
decision  maker’s  insight,  based  on  background  knowledge  and 
sensitivity  to  its  relevance,  is  substituted  for  the 
"structured  decision  approach." 

Often  manv  experts  are  consulted  because  the  problems 
are  too  broad  for  one  individual.  Even  in  narrow  problems, 
the  investment  may  be  too  large  to  rely  on  a  single  decision 

maker  to  derive  all  the  priorities  (Brown,  1968:13).  A 
consensus  of  these  experts  is  needed  to  present  a  unified 


"group"  judgement 


How  the  test  itees  were  derived  should  be 
explai  ned. 

Whet  pilot  procedures  were  used  for  item  analysis 
to  prune  down  the  original  pool  to  the  final  set 
used  for  the  study. 

What  psychometric  scaling  approach  was  selected? 

How  were  the  experts  choosen  -  selective  or  random? 

Are  the  experts  related  to  objectively  defined 
populations  with  measureable  parameters? 

What  are  expected  statistics  regarding  the 
precision  of  results  for  the  sample  size  and 
dispersion  results? 

(Saakman, 1975s 59) 

Saakman’s  point  was  that  mauiy  methods  for  eliciting 
expert  weights  were  unsystematic  opinion  surveys.  Opinion 
surveys  should  fallow  the  American  Psychology  Association 
(APA)  standards  of  rigorous  test  development  and  validation 
<Saakman,  1975:48).  If  needed,  analysts  should  seek 
pyschologist  consultants  to  develop  the  tests  they  require  to 
elicit  these  opinions.  Otherwise,  the  "objective"  surveys 
they  apply  may,  in  fact,  bias  the  results  more  than  the 
interacting  group  they  are  trying  to  avoid. 

Experimental  Comparison.  Analysts  however,  can 
increase  the  validity  of  their  findings  by  reducing  the  bias 
that  originates  in  the  actual  solicitation.  Eckenrode  studied 
variations  of  three  basic  mechanics  of  soliciting  weights 
from  individual  decision  makers:  ranking,  rating,  and 
pairwise  comparison.  His  study  provides  an  excellent  overview 
to  these  three  general  methods.  He  used  Kendall’s  Coefficient 
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of  Concordance  to  measure  agreement  among  the  sets  of  eeights 
produced  by  the  various  methods,  as  well  as  the  agreement 
among  the  decision  makers.  He  found  that  the  methods 
established  very  similar  weights  for  the  three  experiments 
using  factual  data,  general  data,  and  predictive  data.  That 
is,  he  found  that  ranking,  rating,  and  pairwise  comparison 
produced  essentially  the  same  ordering  of  weights  over  a  wide 
range  of  data  types  (Eckenrode,  19653185). 

Eckenrode  found  that  rating  the  objectives  produced  the 
greatest  variance  and  the  narrowest  range  between  weights. 
Eckenrode  considered  small  ranges  and  wide  variance  to  be 
undesirable  characteristics  for  a  method  to  be  used  in 
weighting  multiple  criteria  because  of  lessoned 
discrimination.  As  expected,  he  found  that  the  degree  of 
agreement  appeared  to  decrease  as  one  moved  from  a  relatively 
specific  system  to  a  more  general  problem  (Eckenrode, 
19653187). 

Saaty  reported  about  an  experimental  test  of  the  AHP  by 
Schoemaker  and  Waid  in  1978  to  compare  AHP  with  the  multiple 
regression  approach,  the  multi  -  attribute  utility  approach 
of  Keeney  and  Raifa  and  simple  direct  assessment.  These  four 
methods  differ  in  several  ways:  (1)  They  require  different 
types  of  judgements,  (2)  They  require  different  response 
measures  (ordinal,  interval,  and  ratio),  and  finally,  (3), 
each  method  can  be  applied  in  areas  in  which  the  others  would 
be  limited  (Saaty,  1980:246). 


In  tha  ■xparimant,  subiscta  (33>  wara  askad  to  avaluata 


hypothatical  admiaaiona  candidataa  (20  paira),  in  pairwiaa 
conpariaona,  uaing  four  attributaa  onlyi  quantitativa  SAT, 
varbal  SAT,  high  achool  cunulativa  avaraga,  and  an  indax  o4 
axtra-curricular  activity.  Thair  choicaa  batwaan  thaaa  20 
paira  of  candidataa  waa  conaadarad  tha  "corract"  la  at  to 
compare  -four  dif-farent  aiathoda  o-f  “automated”  decision  making 
functional  the  AHP,  Regreaaion,  Multi  Attribute  Utility,  and 
Direct  Asseasment. 

To  derive  theae  functions  the  subjects  were  asked  for 
further  judgements  (such  as  comparing  the  attributes 
directly)  and  four  linear  additive  representations  were 
constructed.  Below  are  the  correct  predictions  and  Pearson 
product-moment  correlations  between  the  predictexi  and 
observed  strengths  of  preference: 


AHP . 

Regression  ....... 

Mul ti -Attribute  Utility. 
Direct  Assessment.  .  .  . 


Correct  Correlation 


.  .  84X  .72 

.  .  57X  .19 

.  .  SbX  .75 

.  .  B4X  .77 

(Saaty,  1980:246) 


The  advantages  of  each  method  determine  its 
appropriateness  for  a  given  application.  Eckenrode  found  that 
direct  assessment  required  the  least  amount  of  time 
(Eckenrode,  1965:186).  Subjects  are  just  asked  to  rank  or 
rate  a  single  column  of  criteria.  So  when  time  is  a  critical 
element  and  there  are  simple  relationships  between  the 
criteria,  direct  assessment  may  be  the  appropriate 
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l■•thodolgy.  Whila  it  is  also  likaly  that  an  axpart  can 
Bstifiata  tha  situation  diractly  and  may  not  do  batter  with 
AHP  approach;)  tha  AHP  gives  a  measure  of  consistency 
which  is  not  available  in  direct  methods  (Saaty,  1980364) 
Consistency  (transitivity)  is  not  required  among 
preferences  in  tha  AHP,  while  the  construction  of  a  utility 
'^UT'ction  by  multi —attribute  utility  theory  (HAUT)  requires  a 
transitive  preference  relation.  Both  of  these  methods  deal 
with  the  subjective  bias  of  expert  judgement.  The  judgenient 
in  terms  of  a  single  subcriterion  affects  the  measures  of 
othe!r  "independent"  criteria.  This  results  in  a  vector  that 
has  bias  between  its  components,  sometimes  double  counting 
the  importance  of  a  subcriterion  (Saaty,  1980:237). 

□ne  of  MAUT’s  strangest  advantages  is  that  it  has  a  well 
developed  methodology  for  handling  situations  with  risk  and 
encompasses  utility  functions  that  are  not  linear  (Keeney  and 
Raiffa,  1976:34).  The  main  disadvantage  of  MAUT  is  that  the 
elicitation  process  is  laborious.  For  repetitious  decision 
making,  MAUT,  with  its  "standard”  utility  function  would  be 
more  advantageous.  However,  in  practice  the  utility  function 
changes  often  and  requires  constant  revisions.  Coty  explained 
that: 

MAUT  gives  a  complete  orderings  of 
alternatives  but  at  some  cost  in  time  spent 
establishing  the  functional  relationhips  among 
performance  indices.  The  temporal  nature  of  some 
state  vector  elements  may  also  cause  some 
di f f icul ty. 


(Coty,  19B0:AA-15) 


I"  IP  I  u 


Thus,  oparationally,  MAUT  looses  its  advantage  over  AHP 
because  of  its  "detailed  elicitation  to  establish  form  and 
constituents  of  scaler  social  choice  function"  (DeWisplcare 
and  Sage,  1981:16).  AHP,  on  the  other  hand,  can  perturb 
judgments  within  the  hierarchy  to  get  a  new  set  of  priorites, 
without  reconstructing  the  utility  function  for  each  period 
of  time.  (Saaty,  1980:246). 

The  regression  techniques  are  strongest  in  well  defined 
situations  that  have  precise  measures.  However,  in  such  areas 
as  technology  forecasting,  these  measures  are  often  elusive, 
and  the  application  of  regression  is  inappropriate  and  gives 
distorted  results.  Saaty  was  critical  of  regression 
techniques  (and  equal  weighting)  because  they  either  ignore 
relevance  by  using  arbitrary  statistical  objectives,  or  focus 
on  other  subcriteria  rather  than  the  global  criterion  (Saaty, 
1980:237).  MAUT  and  AHP  are  better  tools  for  these  "fuzzy" 
appl i cati ons. 

These  results  aipply  directly  to  this  thesis.  The 
original  Battel le  survey  used  direct  ratings  to  elicit  the 
experts’  opinions.  Eckenrode  criticized  ratings  as  a  poor 
discrimating  tool  because  of  the  resulting  small  range  and 
wide  variance  between  judges.  This  thesis  uses  pairwise 
comparisons  to  elicit  the  opinions.  Eckenrode  found  this 
method  more  satisfactory  because  of  the  smaller  variance 
between  judges  and  wider  range  of  the  priority  vector. 
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V«;iui>  Theory*  The  overall  scores  in  the  AHP  can  be 
considered  as  the  valuations  o-f  an  additive  value  -function 
(Kaaienetzky,  1982:705).  Because  the  AHP's  -foundation  is 
elicitation  of  additive  value  functions,  more  discussion  is 
necessary  on  the  concepts  involved. 

Value  theory  establishes  the  fundamental  conditions  for 
construction  of  order  preserving  representation  of  values  on 
a  set  of  alternatives.  There  is  a  distinct  line  between  value 
functions  and  utility  functions.  Preference  representation 
under  certainty  is  referred  to  as  a  value  function,  and 
preference  representation  under  uncertainty  is  referred  to  as 
a  utility  function  (Keeney  and  Raiffa,  1976).  This  research 
focuses  on  value  functions  because  decision  under  certainty 
is  the  underlying  assumption  in  the  application  of  the  AHP. 

A  value  function  relates  the  value  of  given  attributes. 
It  says  that  if  attribute  b  is  preferred  to  attribute  c  then 
the  value  of  b  is  greater  than  the  value  of  c,  and  vice 
versa.  This  relationship  can  be  represented  functionally: 

b  ^  c  <»>  V(b)  >  V(c) 

Value  functions  are  classified  in  two  general  forms: 
multiplicative  and  additive.  The  additive  form  of  a  value 
function  has  more  restrictive  conditions  and  is  the  type  of 
value  function  assumed  in  the  Al^.  It  allows  the  value  of  a 
given  objective  to  be  the  sum  of  the  values  of  its 
attributes.  For  the  single  decision  maker  the  additive  form 
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can  b«  •xprMMdt 


V(Xl,X2,...,Xn)  «  ^Vi(Xi) 

(Ksaniby  and  Raif-fa,  1976:116) 

It  can  b«  shown  that  if  tha  V  is  bounded,  this  equation 
can  be  rewritten: 

V(X1,X2,  ...,Xn>  »  ^MiVi(Xi> 

t 

Where: 

Vi,  i  s  l,...,n  is  scaled  from  zero  to  one 

Zwi  :=  1 

(Kamenetzky,  1982:705) 


An  additive  value  function  for  a  group  is  comparable  to 
the  additive  value  function  for  an  individual.  The  difference 
is  that  the  attributes  for  the  group  decision  maker  are  the 
values  the  individual  members  ascribe  to  a  given  set  of 
attributes.  The  group  decision  maker  will  be  referred  to  as 
the  supra  decision  maker,  and  she  may  be  benevolent  dictator 
or  the  group  itself  through  interaction.  The  additive  group 
value  function,  Vg,  may  be  expressed: 

Vg(Xl,X2 . Xn)  =  Vgi  CVi  ( Xi  )  D 

L 

Where: 

Vg  »  Supra’ s  Value  Function 

Vgi  =  Supra’ s  Value  of  Individual  i’s  Values 

(Keeney  and  Raiffa,  1976:525) 


A  necessary  and  sufficient  condition  for  an  additive 
value  function  is  mutual  preferential  independence  (I1PI).  MPI 
is  implied  if  every  pair  of  attributes  is  preferentially 
independent  of  its  complement  (F’PI).  For  a  group  value 
function,  the  attributes  £Vi,Vj>  are  preferentially 
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indepBnd0nt  of  thair  complaimint  Vli  pforalli^j,  n>3 
(Kaanay  and  Raif-fa,  1976:524). 

Prafarantial  indapandenca  implias  for  any  tao 
individuals,  i  and  j,  i-j,  if  all  othar  n-2  individuals  ara 
indiffarent  betaaan  a  pair  of  consaquancas,  than  tha  group 
prefarence  depends  only  on  the  preferencas  of  individuals  i 
and  j.  In  particular,  the  group  value  function  should  not 
depend  on  the  level  of  preferences  of  the  othar  individuals 
(Keeney  and  Raiffa,  1976:525). 

For  example,  consider  a  three  person  group  and  start 
with  the  consequences,  X’  and  X**.  Now  suppose  V(X’)  =  V(X") 
by  individual  3.  Preferential  independence  implies  that  the 
group  preference  of  X’  relative  to  X"  should  not  depend  on 
whether  individual  3  considers  X’  and  X“  good  or  bad.  The 
group  ranking  of  X’  and  X"  should  depend  only  on  the  opinions 
of  individuals  1  and  2. 

If  the  values  of  the  attributes  are  not  independent, 
then  they  cannot  be  added  together.  Their  representation  is 
limited  to  the  ordinal:  >,  <,  =,  scale: 

1)  Utilities  experienced  on  the  set  of  alternatives 
allow  for  numerical  representations  on  the  ordinal 
seal e. 

2)  Additive  representation  implies  independence  of 
the  utilities  involved. 

(Baas  and  Kwakernaak,  1977:47) 

Therefore,  attributes  must  be  NPl  to  have  an  additive 
value  function,  otherwise  the  preference  representation  is 
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ordinal.  This  requiramant  is  not  mads  claar  in  darivation  of 
tha  AHP.  Qna  of  tha  raasons  nay  ba  that  astablishing  MPI 
would  aliaxnata  tha  advantaga  that  tha  AHP  has  ovar  MAUT  and 
HAVT  -  tha  burdansona  ali citation  procass. 

Crltiaua/Coppar i son.  Saaty  notad  that  tha  scoras  ara 
indapandent,  but  tha  raquiranant  for  prefarantial 
indapandanca  has  not  recaivad  a  "passing  mantion"  in  most 
applications  of  tha  AHP  (Kamanatzky,  19B2s705).  Kanenatzky 
compared  tha  AHP  with  two  other  methods  that  more  rigorously 
establish  HPI:  the  midvalue  splitting  technique  and  the 
lock-step  procedure. 

In  tha  lock  step  procedure,  the  decision  maker  is  asked 
to  identify  points  in  the  consequence  space  among  which  he  is 
indifferent.  By  equating  the  value  of  these  points  and 
arbitrarily  assigning  the  unit  of  measurement,  tha 
uni  dimensional  value  functions  and  the  overall  value 
functions  are  assessed  simultaneously.  The  resulting  value 
function  is  measured  on  an  ordinal  scale. 

In  the  fflidvalue  splitting  technique,  as  in  the  AHP,  the 
single  attribute  value  functions  are  assessed  independently. 
Also,  like  the  AHP,  this  technique  consists  of  two  separate 
steps:  (1)  assessment  of  the  unidimensi onal  value  functions 
and  (2)  assessment  of  the  weighting  constants  (Kamenetzky, 
1982:706) . 

In  each  of  these  two  steps,  there  are  some  perceived 
advantages  and  disadvantages  of  the  mid value  splitting 
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tachniqiM  conparafl  to  th«  AHP.  Th»  iiidvaluo  method  asmunms 
that  the  decision  maker  can  order  the  differences  in  the 
strength  of  preference  among  alternatives..  In  contrast,  the 
AHP  assumes  that  the  decision  maker’s  preferences  can  be 
measured  on  a  ratio  scale. 

Kanenetzky  favors  the  AHP  when  there  are  a  small  number 
of  alternatives  for  the  estimation  of  the  value  functions, 
Wj/i  or  Vi(Xji>,  for  the  following  reasons: 


1)  Pairwise  comparison  judgments  are  easy  to 
elicit. 

2)  The  weights  Wj/i  synthesize  the  information 
contained  in  all  possible  pairwise  comparisons 
among  the  m  alternative  actions  with  respect  to  the 
criterion  Ki .  The  redundancy  of  the  information 
contained  in  these  ffl(m-l)/2  pairwise  comparisusns 
contributes  to  the  robustness  of  the  estimates. 

3)  The  AHP  allows  for  inconsistency  of  the 
pairwise  judgments  and  provides  a  measure  of  such 
i neons i stency. 

(Kamenetzky,  1782:706) 


Kamenetzky  favors  the  midvalue  splitting  technique  with 
regard  to  assessing  the  weights  of  the  criteria  for  the 
following  reasons: 


1)  The  assumption  of  mutually  independent 
attributes  is  made  explicit  and  its  validity  is 
tested. 

2)  The  question  asked  in  the  AHP  for  the 
elicitation  of  the  scaling  constants  is  too  vague: 

"What  is  the  relative  importance  of  criterion  Ki 
with  respect  to  criterion  Kk?" 

(Kamenetzky,  1982:707) 
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The  ratio  Wi/Wk  is  tha  marginal  rate  of  substitution  of 
Wj/i  for  Wj/k,  Nhich  depends  on  the  scale  on  which  the  Wj/i 
and  Wj/k  are  measured.  As  a  result,  a  better  question  would 
bes 


“Considering  that  the  Wj/i  are  scaled  such  that  the 
sum  of  all  Wj/i  -  1,  what  is  the  ratio  of  the 
increase  in  Wj/i  that  would  be  necessary  to 
compensate  for  a  given  decrease  in  Wj/k  to  the 
given  decrease  in  Wj/k?“ 

(Kamenetzky,  1982:709) 

Kamenetzky  found  contradictory  opinions  asserting 
whether  decision  makers  will  provide  the  same  answers  to 
either  of  the  above  questions.  For  example,  Garielli  and  Von 
Winterf eldt ’ s  results  showed  the  weight  estimates  were 
insensitive  to  changes  in  the  range  of  the  attributes 
(Kamenetzky,  1982:710). 

They  hypothesized  that  people  can  assign  importance  to 
attributes  without  specified  alternatives  and  ranges  because 
they  have  some  plausible  set  of  alternatives  and  ranges  in 
mind,  when  judging  importance.  Perhaps,  Kamenetzky  argues, 
this  is  why  another  study  found  no  difference  among  seven 
different  methods  of  collecting  attribute  weights,  with  five 
of  the  methods  assessing  weights  for  linear  additive  models 
with  no  reference  to  individual  attribute  ranges  (Kamenetzky, 
1982:707)  . 

Keeney,  however,  warns  that  questions  need  to  have 
attribute  ranges.  "A  common  error  made  in  many  studies  is  to 
ask  which  of  several  attributes  is  most  important, 
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indapandant  of  thair  rangas"  (Keanay,  1977s 286) 


Saaty  beliavas  tha  phrasing  should  reflect  the  proper 
relationship  batMaen  elanents  in  one  level  with  the  property 
in  the  next  higher  level.  He  gave  tha  following  exaaples 


If  the  el aments  are  dominated  by  the  property 
rather  than  vice  versa,  ask  how  much  more  strongly 
the  element  is  dominated,  affected  by,  and  so  on  , 
this  property.  In  projecting  an  outcome,  ask  which 
element  is  more  likely  to  be  decisive  or  to  result 
in  the  outcome. 

(Saaty,  1982:77) 


Kamenetzky  inclines  toward  Keeney's  view.  He  feels 
Saaty* s  question  is  so  vague  that  the  weights  Mi  obtained  by 
asking  this  question  may  not  reflect  the  decision  maker’s 
preferences  accurately.  This  is  not  true  in  the  midvalue 
splitting  method  because  the  way  in  which  the  weights  are 
estimated  insures  that  the  resulting  value  function  will  be 
consistent  with  at  least  some  of  the  decision  maker’s 
preferences,  i.e. ,  those  involving  the  carefully  chosen  n-1 
pairs  of  alternatives  (Kamenetzky,  1982:708). 

AHP-I*li dval  ue  Combi  Kamenetzky  suggests 

combining  the  best  of  the  midvalue  splitting  technique  and 
the  best  of  the  AHP  in  four  steps: 


1)  Check  whether  attributes  are  mutually 
preferentially  independent  and  if  there  are 
sufficient  conditions  satisfied  for  the  existence 
of  a  measurable  value  function.  The  attributes  must 
be  mutually  preferentially  independent  in  order  to 
apply  any  of  these  techniques.  If  there  is  not  a 
measurable  value  function,  it  cannot  be  asserted 
that  the  preferences  of  the  decision  maker  can  be 
represented  by  an  interval  scaled  additive  value 
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function.  Consequsntly,  naither  can  it  be  asserted 
that  they  can  be  represented  by  ratio  scaled  units. 

2)  Obtain  the  relative  standing  of  each  action 
Mith  respect  to  each  criterion  Wj/i  by  the  midvalue 
splitting  technique. 

3)  Obtain  a  nem  set  of  weights  Sj/i,  by 
scaling  the  Wj/i  so  that  fflax(j)Sj/i  =  1,  and 
min(j)Sj/i  =  0  by  using  the  transformation: 

Sj/i  =  (Uj/i  -  nin(j>Wj/i)  /  (max(j>Wj/i  - 
min ( j ) Wj/i ) 

j  —  1  f  m  •  m  f  nif  i  =  l.....P. 

(Kamenetzky,  1982:709) 

If  the  denominator  above  equals  zero,  attribute  Ki 
should  not  be  considered  because  it  does  not  discriminate 
among  the  alternatives  being  considered.  The  Sj/i  obtained 
from  this  transformation  will  not  be  expressed  on  a  ratio 
scale  anymore,  but  they  will  still  constitute  the  valuations 
of  p  uni dimensi anal  value  functions.  Since  a  ratio  scale  has 
all  the  properties  of  an  interval  scale  and  an  interval  scale 
is  unique  up  to  a  linear  traknsf ormation,  these  value 
functions  can  be  given  a  strength  of  preference 
interpretation. 

The  transformed  weights  for  the  actions  are  then  used  to 
define  the  value  function,  Vi(Xj)  =  Sj/i  which  is  weighted 
and  summed  over  all  the  criteria.  The  criterion  weights  are 
determined  by  making  pairwise  comparisons  between  these  value 
functions  using  the  AHP.  The  normalized  eigenvector 
establishes  the  weights.  Kamenetzky  covers  an  alternative 
method  to  establishing  the  weights  by  "restricting  the  use  of 
the  additive  form  to  a  smaller  region  of  the  consequence 
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space,"  that  is,  he  reduces  the  number  o-f  ccMiparisans  to  be 
evaluated. 

Kamenetzky  dc3es  not  claim  that  the  overall  measure  of 
preference  obtained  by  this  hybrid  method  is  ratio-scaled,  as 
it  is  in  the  AHP.  This  measure  is  intended  to  be  used  as  an 
ordinal  measure  of  preference  in  order  to  facilitate  the 
process  of  eliciting  single  attribute  preferences  and  scaling 
constants.  It  is  assumed  that  the  decision  maker  can  provide 
judgments  of  preference  on  a  ratio  scale.  Consequently,  to  be 
rigorous,  it  should  be  checked  that  it  meets  the  sufficient 
conditions  for  a  ratio  scaled  additive  value  function. 
Although  the  conditions  for  an  interval  scaled  additive  value 
function  have  been  listed  (Dyer  and  Sarin,  1979:810),  no  one 
has  determined  the  sufficient  conditions  for  the  existence  o-f 
a  ratio  scaled  additive  value  function  (Kamenetzky, 

1982:712) . 

Kamenetzky  summarizes  the  differences  in  his  hybrid  AHP 
from  the  standard  AHP: 


1.  The  assumption  of  mutual  preferential 
Independence  of  the  attributes  is  made  explicit  and 
its  validity  is  tested.  Also,  it  is  tested  whether 
the  sufficient  conditions  for  the  existence  of  an 
interval  scaled  additive  value  function  are 
satisfied. 


2.  The  question  “What  is  the  relative  importance  of 
criterion  Kk  with  respect  to  criterion  Kh?”  is 
replaced  by  a  more  specific  question  concerning  the 
relative  preferability  of  two  well  defined 
al ternat i ves. 


3.  The  Wj/i  are  scaled  from 
value  functions  of  actiansl 


zero  to  one  Cused  as 
(Kamenetzky,  1982:712) 
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requiring  the  outliers  to  do  extra  work  (Saakman,  1975:45). 
Helner  explains  that  requiring  respondents  to  justify 
“outlier"  opinions  helps  to  solidify  the  multi -peaked 
characteristic  of  Delphi  rather  than  force  an  artificial 
consensus: 


The  effect  of  placing  the  onus  of  justifying 
relatively  extreme  responses  on  the  respondents  had 
the  effect  of  causing  those  without  strong 
conviction  to  move  their  estimate  closer  to  the 
median,  while  those  who  felt  they  had  good  argument 
for  a  deviation! St  opinion  tended  to  retain  their 
original  estimate  avid  defend  it. 

(Wheelwright,  1980:282) 


Saakman  also  critiqued  the  insufficient  reliability,  the 
sensitivity  to  the  ambiguity  in  questions,  and  the  difficulty 
in  assessing  the  degree  of  expertise  to  determine  the 
"decision  makers."  Although  these  complaints  are  valid 
criticisms,  they  are  relative  to  the  entire  field  -  the 
Delphi  method  should  be  judged  xn  terms  of  the  available 
alternatives.  The  same  objections  apply  even  more  critically 
to  the  less  systematic  methods  of  forecasting  (Wheelwright, 
1980:280) . 

The  author  shares  E.S.  Quade's  biggest  criticism.  Delphi 
is  cumbersome.  It  requires  several  weeks  to  receive  responses 
and  the  iterative  questionnaires  can  be  tedious  (Quade, 
1968:324).  This  may  be  one  of  the  reasons  the  feedback 
results  were  very  difficult  to  ascertain  in  this  research; 
the  subjects  were  all  high  level  defense  experts  who  were 
impatient  to  repetitive  surveys. 
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Saaty  compared  the  Delphi  approach  to  the  AHP  and  found 
the  fallowing  differences: 

1>  Anonyflious  versus  operating  group  discussion. 

2)  Adjustment  in  a  series  of  rounds  versus  dynamic 
discussion. 

3)  Questionnaire  versus  hierarchy  structure  as  a 
basis  for  judgments. 

4)  Statistical  and  quantitative  analysis  versus 
qualitative  analysis. 

(Saaty,  1980:69) 

Both  processes  improve  the  quality  of  judgments,  but 
Saaty  says  the  AHP  better  fits  the  human  cognitive  style 
because  it  breaks  the  judgment  into  its  elements.  Saaty 
further  claims  that  the  group  has  better  confidence  in  the 
relevance  of  their  judgments  because  the  group  determines  the 
important  set  of  variables  (Saaty,  1980:70). 

Del  phi  _Cri  ti  ci  snii  Not  all  the  research  has  been 
positive  regarding  feedback.  Gustafson  found  that  written 
feedback  appeared  to  lead  to  a  reduction  in  the  quality  of 
estimates  in  his  experiment  about  estimating  population 
heights.  He  speculated  that  written  feedback  without 
clarification  may  lead  to  distortion  and  recommended  further 
experimentation  on  the  effects  of  unelaborated  feedback  on 
judgemental  tasks  (Gustafson  and  others,  1973:280). 

Saakman  has  criticized  the  Delphi  method  on  several 
accounts,  particularly  for  what  he  considers  the  artificial 
consensus  brought  about  by  the  "bandwagon  appeal "  and 
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itaration,  including  the  first.  The  group  is  than  "fed  back” 
the  reasons  but  not  the  statistics  (median,  etc).  The  process 
is  repeated  until  the  individual  judg«ients  stabilize. 

OFC  is  particularly  appropriate  to  the  AHP  questionnaire 
because  the  reasons  for  the  individual's  priority  vector  can 
be  collected  beginning  on  the  second  iteration  (after  the 
vectors  have  been  computed  from  the  comparison  matrix).  The 
number  of  reasons  mould  be  well  within  reason  and  could  be 
assimilated  into  the  decision  maker's  thought  processes  as  he 
revises  his  comparison  matrix. 

Press  explained  the  responses  of  the  group  members  as  a 
function  of  the  feedback.  The  student— t  distribution  was  used 
to  show  the  systematic  convergence  of  the  reasons  (Press, 
1979c: 3).  He  found  that  the  OFC  judgements  »<iere  more 
systematic  and  stable  than  the  control  group  which  received 
no  feedback.  Since  the  feedback  is  "qualitative,"  Press 
claims  the  procedure  does  not  induce  an  artificial  consensus 
on  the  group  (Press,  1979b: 1). 

BbElnSsIebiv  Saaty  mentions  that  it  is  possible  to 
elicit  the  hierarchy  and  judgements  by  questionnaire  (Saaty, 
1980:35).  If  he  were  to  do  this  using  anonymity,  controlled 
feedback,  and  statistical  group  responses,  he  would  be 
applying  the  Delphi  technique  to  the  AHP.  The  Delphi 
technique  is  a  well  established  methodology  to  collect 
opinions  remotely,  such  as  mailed  questionnaires.  Many 
researchers  consider  it  better  than  interacting  groups. 
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is  multi -peaked,  rather  than  single-peaked.  Most 
cases  Delphi  methods  have  been  applied  to  in  the 
past  are  these  single-peaked  straight  forwardly 
factual  issues,  a  prospect  more  than  likely  in  many 
spheres  of  the  value  area. 

(Rescher,  1969e6> 

ESSdIidckj.  There  are  many  variations  of  the  Delphi 
approach,  primarily  varying  the  type  of  feedback  given  to  the 
respondents.  As  long  as  the  feedback  involves  some  relatively 
precise  summary  of  the  previous  round,  the  form  of  feedback 
does  not  have  much  affect  on  the  variance  between  judges 
(Dalkey  and  others,  1970:28). 

□ne  version  of  Delphi  returns  only  the  percentile  score 
to  each  participant.  Thus,  the  respondent  only  knows  the 
general  direction  of  the  group  consensus,  rather  than  the 
actual  mean  or  median.  Dalkey’s  experiments  showed,  however, 
that  percentile  feedback  appeared  to  be  slightly  less 
effective  with  respect  to  average  error,  than  medians  and 
quart iles  (Dalkey  and  others.  1970:28,3). 

9y*l.Lfe)t_Eegdback_Cgntrol^j_  James  Press  proposed  a 
Delphi  variant  which  he  called  Gkiality  Feedback  Control  (QFC) 
to  help  elicit  the  value  judgements  and  recommended  decisions 
of  groups  (Press.  1979b; 1).  The  method  shares  two 
distinctions  of  Delphi:  anonymity  and  controlled  feedback. 

QFC  differs  from  Delphi  in  that  it  does  not  return 
statistical  group  responses.  While  Delphi  collects  reasons 
for  deviant  responses  on  the  second  and  subsequent 
iterations.  QFC  collects  reasons  for  each  answer  on  every 
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Delphi  avoids  the  dys-f unctional  group  interactions.  The 
separation  of  the  participants  helps  to  insure  that  their 
judgements  Mill  not  be  influenced  by  social  pressure  towards 
conformity  or  other  aspects  of  small  group  behavior,  such  as 
dominant  individuals  or  irrelevant  communication  (Dalkey, 
1969b:5}.  The  approach  is  broken  into  the  following  phases: 


1.  The  experts  are  asked  for  their  judgements  on  a  series  of 
mailed  questions. 

2.  The  median  and  quartile  scores  for  each  question  are 
returned  to  the  respondents.  On  those  items  on  which  there 
is  no  general  agreement, the  experts  are  asked  to  state  their 
reasons  for  their  divergent  estimates. 

3.  Process  2  is  repeated  1-3  times  with  the  explanations  for 
the  outliers.  This  reduces  the  quartile  range  without 
pressuring  the  respondents. 


Delphi’s  focus  is  the  methodology  to  collect  these 
opinions.  Unlike  many  forecasting  methods,  Delphi  does  not 
have  to  produce  a  single  answer  as  its  output.  Instead  of 
reaching  a  consensus,  the  Delphi  approach  can  leave  a  spread 
of  opinions,  since  there  is  no  particular  attempt  to  get 
unanimity.  Rescher  reports  this  feature  makes  Delphi  more 
appropriate  for  exploring  group  examination  of  values,  rather 
than  Delphi’s  traditional  role  to  assess  expert  opinion  about 
factual  questions: 


Rather  thatfi  serving  as  a  tool  for  discovering 
or  forging  a  group  consensus,  Delphi  can  provide  a 
technique  for  discovering  subgroups  of  variant 
opinion  in  a  group  -  when  the  pattern  of  responses 
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survsys  are  also  becoining  nore  responsivs.  It  is  foreseeable 
that  in  the  not  so  distant  future,  a  national /mot Idwide 
conputer  network  will  link  everyone,  and  surveys  will  have 
the  sane  response  time  as  interactive  groups. 

There  have  been  few  studies  on  applying  the  AHP  by 
survey.  This  study  will  apply  the  AHP  via  nailed 
questionnaires  to  examine  this  aspect  of  the  AHP.  The  process 
used  to  solicit  the  opinions  will  be  a  version  of  the  Delphi 
technique  first  developed  by  Rand  Corporation  in  the  1950s. 

Discyssi.gn^  The  Delphi  approach  is  among  the  most 
common  methods  for  qualitative  forecasting  (Wheelwright, 
1980:298).  It  is  a  solicitation  method  that  iteratively  uses 
surveys  and  statistical  feedback  to  achieve  a  group  decision. 

Both  Delphi  and  the  AHP  improve  the  quality  of 
judgments,  but  Saaty  claims  thatthe  AHP  better  fits  the  human 
cognitive  style  because  it  breaks  the  judgment  into  its 
elements.  Saaty  also  claims  that  the  group  has  better 
confidence  in  the  relevance  of  their  judgments  because  the 
group  determines  the  important  set  of  variables  (Saaty, 
1980:70) . 

Saaty  sees  the  Delphi  approach  as  the  end  product.  It  is 
not.  It  is  a  technique  identified  by  the  following  three 
aspects  which  would  be  an  excellent  tool  to  apply  the  AHP  by 
survey: 

1 )  Anonymous  response. 

2)  Iterative  and  controlled  feedback. 

3)  Statistical  group  response. 

(Dal  key.  1969b: v) 


amount  (Rowsm,  1974:280).  This  was  ths  result  that  Battelle 
fcxjnd  when  they  performed  a  sensitivity  analysis  on  the  top 
level  weights.  The  weights  could  change  by  a  factor  of  four 
and  still  have  negligible  effect  on  the  results  (Rice, 
1983:2-86) . 

With  regard  to  weights  on  individual  opinions,  one  is 
not  necessarily  seeking  a  change  in  the  results,  only  that 
the  results  are  more  representative  of  the  group.  Keeney 
insists  that  weights  are  necessary  to  avoid  intransitive 
results  and  Rowse  found  that  personal  weights  resulted  in 
more  accurate  group  responses  than  interacting  groups. 
Therefore,  this  study  will  pursue  weighting  individuals  to 
achieve  a  more  valid  representation  of  the  group  opinion. 

Delphi.  Saaty  maintains  that  in  “seeking  consensus," 
it  is  preferable  that  the  judges  interact"  (Saaty,  1980:68). 
He  considers  the  single  input  of  interacting  judges 
preferable  over  deriving  a  mathematical  mean  input  (Saaty, 
1980:32).  The  aggregation  methods  that  require  all  the 
decision  makers  to  be  physically  collocated  are  common  for 
small  sample  sizes  or  homogeneous  groups.  However,  the 
decision  makers  who  have  the  diverse  backgrounds  necessary 
for  research  and  development  questions  are  often  dispersed 
across  wide  regions.  Requiring  them  to  interact  physically 
would  restrict  the  sample  size  and  the  caliber  of  "expert 
decision  maker"  solicited. 

Besides  allowing  a  wider  dispersion  of  participants. 
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individual  inputs  (Saaty,  1980368). 

Rowse  par-forned  an  axparimant  where  he  evaluated  the 
accuracy  o-f  group  decisions  as  a  function  of  different 
weighting  schemes.  Interacting  group  responses  were  compared 
to  five  weighting  schemes.  The  five  schemes  were  equal 
weights,  peer  assigned  weights,  self  assigned  weights,  group 
assigned  weights  (a  combination  of  peer  and  self  weights), 
and  average  peer  and  self  weights.  All  his  methods  used  the 
mean  response  after  weighting.  He  measured  the  accuracy  of 
the  group  answers  by  the  least  squared  error  from  the  real 
answer  and  the  group  reliability  by  the  variance  of  the 
estimates  across  methods  a^id  averaged  over  five  questions. 

He  found  no  differences  between  the  five  weighting 
methods,  but  all  the  weighting  methods  did  better  than 
interacting  groups.  Perhaps  the  fact  that  all  the  techniques 
were  reliable  across  similar  questions  may  mean  that  it  is 
possible  to  effectively  predict  the  optimal  range  of 
expertise.  This  would  be  a  very  difficult  task  for  the  supra 
decision  maker  to  determine  the  optimal  range  of  expertise  of 
his  staff.  This  estimate  of  the  difficulty  was  pointed  out  by 
Keeney  when  he  was  referring  to  interpersonal  value 
comparisons  by  either  ::he  "benevolent  dictator"  or  the  group 
itself  evaluating  the  weights  for  its  own  members. 

Another  reason  there  were  no  differences  between  the 
weighting  methods  may  be  that  the  weights  do  not  have  enough 
affect  on  the  estimates  for  changing  the  range  a  significant 
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Dobbins  rsported  that  ths  quality  o-#  a  rank-ordsrad 
priority  aggregation  can  be  significantly  influenced  by  the 
variation  in  each  judge’s  knowledge  of  each  alternative  he 
chooses  to  rank  (Dobbins,  1980a:60>.  Dalkey  also  found  these 
self  ratings  statistically  improved  the  accuracy  of  group 
estimates  of  historical  facts  (Dalkey,  1969c:2>.  In  Dobbin’s 
research,  he  restricted  the  self-ratings  between  zero  (worse) 
and  one  (best).  Dobbins  multiplied  this  self -expertise  rating 
by  the  alternative  scores  for  the  judge’s  decision  matrix  as 
if  the  self-rating  were  a  multiplicative  weight  (Dobbins, 
1980:60) . 

There  have  been  several  applications  of  using 
interpersonal  weights  to  help  accurately  amalgamate  group 
decisions.  The  “soundness”  may  be  measured  by  relative 
intelligence,  years  of  experience,  past  record,  personal 
involvement  in  the  issue  at  stake,  personal  evaluation,  etc 
(Saaty,  1980:68).  Dobbins  used  self  assessments  between  0 
and  1  which  he  directly  applied  to  the  individual’s  inputs  to 
weight  them  (Dobbins,  1980a: 65).  Similarly,  Horsky  and  Rao 
corrected  for  the  judges'  differential  familiarity  with 
various  alternatives  by  having  the  judge  state  his  confidence 
in  his  judgement  for  each  pair  of  comparisons  (Horsky  and 
Rao,  1984:808).  Saaty  gave  more  weight  when  he  had  high 
confidence  in  the  judgement  of  a  particular  individual 
expert.  When  he  had  low  confidence  he  used  the  mean  of  the 
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that  the  dictator  determine  the  best  and  worst  value  from 
each  individual.  Then  he  must  superimpose  his  own  value 
structure  about  the  relative  desirability  of  the  change  in 
the  value  from  0  to  1  for  individual  A  versus  the  change  in 
value  from  O  to  1  for  individual  B,  etc. 

Me  make  no  pretense  that  interpersonal  utility 
comparisons  are  easy,  but  they  are  often  implicitly 
made  in  group  decisions. . .Not  only  does  the 
decision  maker  have  to  scale  intensities  of 
preferences  for  each  of  the  individuals,  but  the 
process  requires  interlinking  the  scales  of  the 
individuals  and  hence  it  involves  extraneous 
interpersonal  comparisons." 

(Keeney,  1976: 145) 

The  same  type  of  thinking  must  be  followed  in  the 
participating  group  decision  model  by  each  of  the  individuals 
in  the  group.  However,  they  must  also  arrive  at  a  consensus 
for  the  constant  weighting  factors.  Sometimes  this  may  not  be 
possible  and  thus  the  model  could  not  be  used  as  intended. 
There  have  been  several  experiments  to  investigate  the 
factors  involved  in  interpersonal  comparisons. 

^BBiicati ons_of _Nei ghti Qg_Indi yidual Si 
Experiments  have  shown  that  if  the  experts  are  asked  to  grade 
themselves  on  competence,  the  estimates  they  give  are  fairly 
reliable.  On  the  basis  of  this  information,  it  may  be 
possible  to  obtain  better  results  by  using  as  the  group 
consensus,  not  the  median  of  all  responses  to  a  question,  but 
the  median  of  the  fraction  who  declared  themselves  most 
knowledgable  with  regard  to  that  question  (Quade  and  Boucher, 


in-flu«ncBB  arm  at  laast  partially  ovarcona  by  aathmatically 
aggragating  individual  astiaataa.  RoMsa  atatad  that 
nathnatical  conaanaua  ahould  “ba  choaan  ovar  bahavioral 
conaanaua  -for  tha  antira  ranga  of  li  kali  hood  ratioa"  (Rowaa 
and  othara,  19743283).  Thia  ia  vary  banaficial  far 
solicitation  by  survoy,  as  in  this  study,  but  contrasts 
sharply  with  Saaty’s  enphaais  on  intaracting  groups  to 
achiava  a  conaanaua. 

EfiB!!4Sizing_Injiyidyais,.  Keanay  adds  an  auiditional 
condition  to  ArroN’s  fiva  conditions  to  inaura  tha  group 
value  function  is  transitive.  Ha  requires  interpersonal 
conparisons  of  preferences  (Keeney  and  Raiffa,  19763526). 
Keeney  gives  two  sources  that  are  able  to  nake  these 
asseasnents:  a  benevolent  dictator  and  the  group  itself  -  as 
a  participating  model  (Keeney,  1976:144).  The  group  priority 
vector  is  then  a  linear  combination  of  the  individual 
priority  vectors. 

Saaty  calls  this  linear  combination  a  ''synthesis'*  of  the 
individuals: 

When  each  of  several  individuals  does  his  own 
evaluation,  the  separate  results  may  be  compared 
from  standpoints  of  individual  utilities  to  obtain 
a  synthesis  performed  by  an  outside  party  of  what 
the  individuals  would  do  jointly. 

(Saaty,  1980:9) 

Assessing  the  interpersonal  comparisons  is  a  difficult 
task.  Keeney-' s  system  to  determine  scaling  constants  requires 


SrouD  Interactions 

Introdijctign.  Thera  have  been  several  studies  to  show 
the  deleterious  effects  of  interacting  groups  and  the  quality 
of  the  final  “group"  decision.  Huber  found  that  although 
there  is  less  variability  when  there  is  sone  group  discussion 
prior  to  data  collection,  the  move  toward  conformity  was 
temporary  and  the  result  of  social  pressures.  He  found  the 
rate  and  extent  of  convergence  of  expressed  outlook  would 
vary  considerably  from  one  organization  to  another  (Huber, 
1968:488) . 

In  another  study,  VandeVen  looked  at  the  different  group 
types  and  found  the  heterogeneous  groups  (low  correlations  in 
personality  profiles)  produced  a  higher  proportion  of  high 
quality  -  high  acceptance  solutions  than  the  homogeneous 
groups  (VandeVen,  1971:208).  The  TAV  steering  committee  used 
in  this  research  would  be  labeled  as  heterogeneous  because  of 
the  diverse  specialties  represented. 

Perhaps  most  significant  was  Dal  key’s  finding  that  more 
often  than  not,  face  to  face  discussion  tended  to  make  the 
group  estimate  less  accurate;  whereas  more  often  than  not, 
the  anonymously  controlled  feedback  procedures  made  the  group 
estimate  more  accurate  (Oalkey,  1969b: v). 

One  of  the  reasons  for  the  inaccurate  face  to  face  group 
decisions  is  that  group  influences  are  dysf unctional  as  an 
aggregation  tool  (Rowse  and  others,  1974:283).  These 


Th»  AHP  doM  not  roquxrc  linear  value  attributes^  but  it 
does  require  that  all  the  conparisons  in  a  given  natrix  be 
within  an  order  of  magnitude  of  value  of  each  other,  so  that 
the  input  conparisons  will  renain  representative  of  the 
relative  values.  Any  objective  whose  value  exceeds  the  other 
values  in  the  matrix  by  more  than  the  scale  allows  (1..9), 
should  be  removed  from  that  matrix,  otherwise,  the  AHP 
assessment  is  inappropriate. 


This  hybrid  procsdurs  rsliss  on  the  AHP  to  build  the 
uni dimensional  value  functions.  It  combines  elements  of  both 
the  AHP  and  the  midvaluc  splitting  method  to  determine  the 
weighting  constants.  It  uses  the  eigenvector  from  the  AHP  to 
measure  the  preference  function  and  the  midvalue  splitting 
technique  to  compare  hypothetical  alternatives  selected. 

U9rth_AssgssaeQt.  Another  elicitation  technique  that 
requires  additive  value  functions  is  worth  assessment.  It  has 
an  additional  restriction  on  top  of  that  required  by  the  AHP 
in  that  it  requires  linear  value  attributes.  The  similarity 
with  the  AHP  requirements  can  be  seen  in  the  definition  of 
worth  independence: 


1)  The  relative  importance  of  satisfying  sepaurate 
performance  criteria  does  not  depend  on  the  degree 
to  which  each  criterion  has  in  fact  been  satisfied. 

2)  The  rate  at  which  increased  satisfaction  of  any 
given  criterion  contributes  to  the  overall  worth  is 
independent  of  the  levels  of  satisfaction  already 
achieved  on  that  and  other  criteria. 

3)  The  rate  at  which  decision  makers  would  be 
willing  to  trade  off  decreased  satisfaction  on  one 
criterion  for  increased  satisfaction  on  other 
criteria  so  as  to  preserve  the  same  overall  worth 
is  independent  of  the  levels  of  satisfaction 
already  achieved  by  any  and  all  of  the  criteria. 

(Sage.  1977:359) 


These  three  statements  imply  that  the  decision  maker’s 
value  curves  are  linear.  If,  at  the  boundaries  of  decision 
space,  the  curves  lose  their  linearity,  then  worth  assessment 
would  no  longer  be  an  appropriate  method  to  elicit  weights. 


Summary  Group  Intgractigny.  The  entire  point  of 
Delphi’s  feedback  and  Saaty’s  interacting  groups,  is  to 
derive  a  group  decision  free  from  spurious  variance.  Large 
variances  tend  to  cancel  out  individual  inputs  auid  that 
objective  receives  less  emphasis.  As  reported  below,  there  is 
less  support  for  the  group  decision: 


If  an  area  is  important  to  our  needs,  but  there  is 
disagreement  on  implementation,  we  would  have  to 
withhold  action  until  people  develop  a  better 
appreciation  for  the  need  and  can  induce  more 
cohesive  action. . .Where  there  is  disagreement, 
people  will  tend  to  be  dissatisfied  because  they 
don’t  see  realization  of  their  judgments. 
Otherwise,  with  agreement  there  is  greater 
satisfaction.  (Saaty,  1980:30) 


Throughout  the  discussion  of  variance  reduction  and 
achieving  consensus,  it  is  important  to  remember  that  group 
consensus  is  not  a  goal  in  itself.  Siegal  emphasizes  this 
point  in  his  statistics  text  when  he  explains  the  Kendall 
coefficient  of  concordance,  which  is  one  measure  of  consensus 
between  rank  orders: 


It  should  be  emphasized  that  a  high  or 
significant  value  of  concordance  does  not  mean  that 
the  orderings  observed  are  correct.  In  fact,  they 
may  all  be  incorrect  with  respect  to  some  external 
criterion....  It  is  possible  that  a  variety  of 
judges  can  agree  in  ordering  objects  because  all 
employ  the  "wrong"  criterion. ..  .A  higli  degree  of 
agreement  about  an  order  does  not  necessarily  mean 
that  the  order  which  was  agreed  upon  is  the 
"objective"  one.  In  the  behavioral  sciences, 
especially  in  psychology,  "objective"  orderings  and 
"consensual"  orderings  are  often  incorrectly 
thought  to  be  synonymous. 

(Siegal,  1956:338) 
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Saaty  enphasizes  that  the  top  level  weight s  are  the  most 


important  and  require  additional  attention.  The  Delphi  and 
QFC  examination  in  this  thesis  will  be  applied  to  the  top 
1  evel . 


One  must  prioritize  very  care-fully  the  highest 
level  o-f  the  hierarchy  because  it  is  there  that 
consensus  is  most  needed  since  these  priorities 
drive  the  rest  of  the  hierarchy. 


(Saaty,  1980:30) 


Introduction.  When  individual  prafsrances  arc  given 
in  terns  o-f  ordinal  rankings,  the  simplest  torn  of  group 
consensus  is  majority  rule  (Cook  and  Seiford,  1978s 1722). 
Majority  rule  (MR)  determines  the  group  ranking  by  the  number 
of  "votes"  received  by  each  abjective.  This  thesis  will 
extend  the  concept  and  determine  the  cardinal  rankings,  that 
is,  the  priority  vector  from  the  number  of  "votes"  received 
for  each  abjective.  A  survey  of  the  literature  exhibits 
several  pertinent  studies  regarding  majority  rule. 

Olscysslgn^  Kendall  reported  that  the  best  method 
associated  with  the  least  squares  is  to  rank  the  sums  of 
ranks  allotted  to  individuals.  The  sums  of  squares  of 
differences  between  what  the  totals  are  and  what  they  should 
be  if  all  rankings  were  alike  is  a  minimum  when  this  method 
is  used  to  estimate  ranks.  Furthermore,  if  the  ranking 
arrived  at  by  this  method  is  correlated  by  Spearman's  rho 
with  the  actual  observed  rankings,  the  mean  rho  obtained  is 
larger  than  for  any  other  estimated  rankings  (Kendall, 
1955:100).  Kendall  broke  tie  scores  by  ordering  the  tied 
scores  by  their  variance.  That  is,  if  there  were  three  tied 
objective  scores,  the  one  with  the  smallest  variance  between 
judges  would  be  ranked  ahead  of  the  others  (Kendall, 

1955:100,  Cook  and  Seiford,  1982:621). 

Fishburn  explains  the  method  to  establish  a  priority 
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vactor  Mith  this  approach  by  using  value  theory.  Instead  of 
votes  he  adds  the  continuum  of  values.  Assuming  that  the 
value  of  the  whole  equals  the  sum  of  the  value  of  the  patf*ts, 
one  may  write: 

U(Xa  j)  =  Xi  j/(:^Xi  j) 

where  U(Xij>  is  the  group  value  of  objective  ij 

Xij  is  the  individual  value  of  objective  ij 

(Saaty,  1980:238) 

This  thesis  extends  Fishburn’s  technique  to  the  voting 
matrix  to  calculate  an  alternative  aggregation  method.  The 
inputs  are  preferences  with  no  strengths  (O, 1,1/2).  Dobbin’s 
reports  that  the  appealing  advantage  of  this  aggregation 
method  for  voting  matrices  is  that  intransitive  rank  orders 
appear  only  as  indifferent  alternatives  (Dobbins,  1980a:43). 
For  example,  one  judge  votes  the  objectives:  A^B^C,  and 
another  judge  votes  the  objectives:  B^A^.  Their  aggregate 
order  will  be  A=B^C.  The  greater  the  number  of  judges,  the 
more  likely  there  will  be  no  tie  scores  because  of  the  finer 
discrimination  between  elements.  For  each  n-matrix  the  total 
votes  cast  is  equal  to  (number  of  judges) (n) (n-1 ) /2.  The 
priority  vector  derived  from  ordinal  ranks  becomes  more 
continuous  with  more  judges,  or  increasing  n,  and  better 
reflects  the  group  priority  vector. 

This  study  begins  with  the  AHP  ratio  comparisons  and 
converts  them  into  ordinal  preference  relations  in  order  to 
apply  majority  rule.  There  are  many  majority  rule  methods 
that  use  slightly  different  techniques  to  aggregate  ordered 
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ranks  of  prefsrsnces.  Dobbins  camparad  tha  fol lowing  majority 
rula  techniquas: 


1)  Borda's  tiathod  of  Harks 

2)  Adjustad  Borda  Mathod  of  Marks 
3>  Condorcat*s  Critarion 

4)  Black's  Sifflpla  Majority  Procadura 

5)  Dodgson's  Method  of  inversion 

6)  Copeland's  Majority  Rule 

7)  Shannon  and  Svestka's  Majority  Rule 
B)  Black's  Single  Peaked  Preference 

(Dobbins,  19B0a:2S) 

Dobbins  found  that  the  Shannon  method  allowed  him  to 
complete  the  most  of  the  following  requirements  for  his 
computer  model . 


1.  Accepts  Ordinal  or  Cardinal  Data 

2.  Accepts  Intransitive  Rank  Orders 

3.  Accepts  Partial  orders 

4.  Accepts  Weak  rank  orders 

5.  Accepts  An  odd  or  even  number  of  judges 

6.  Majority  winner  is  ranked  in  first  place 

7.  Method  will  balance  A>B  in  an  individuals 
preference  ordering  by  8>A  in  some  other 
individuals  ordering  as  long  as  there  are  no  other 
alternatives  between  A  and  B  in  either  ordering. 

(Dobbins,  19B0a:28} 


This  study  will  also  use  the  Shannon  Majority  Rule 
method  for  the  same  reasons  that  Dobbins  lists  above.  This 
thesis  will  extend  the  Shannon  Majority  Rule  method  one  step 
and  interpret  the  ordinal  output  ranks  as  cardinal  ranks  to 
compare  them  to  the  vectors  derived  by  the  AHP. 

Ellsburg  noted  that  a  transformation  of  value  numbers  to 
the  ordered  rank  would  not,  in  general,  "preserve  the 
equality  or  give  inequalities  among  differences  between 
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utility  numbBTs"  (Ellsburg,  1954:272).  This  will  bs  Bxsiiinsd 
in  this  thBSis  by  comparing  thB  priority  vBCtors  dBrivBd  by 
both  thB  AHP  ratio  scalB  and  tha  voting  matrix  ordinal  scale. 

Transitivity.  Many  studiBs  havB  bBBn  dirsctad  toward 
datBrnining  tha  probability  o-f  achiaving  an  intransitiva 
aggragata  rank  ordar  through  a  majority  rula  method,  and  what 
conditions  are  required  ior  a  usable  majority  rule  decision. 
Dobbins  describes  two  of  the  conditions  which  are  actually 
restatements  of  Arrow’s  properties: 

1)  Scoring  constants  should  logically  be 
have  autonomy,  be  non-reversed,  and  have 
non-dictatorship. 

2)  The  social  welfare  function  Cgroup  priority 
vector]  should  exhibit  raonotonicxty  and 
faithfulness. 

(Dobbins,  19B0a:20) 

An  accepted  limitation  of  the  unrestricted  use  of  many 
majority  rule  methods  is  that  the  aggregated  results  are 
intransitive.  Dobbins  examined  the  efforts  to  develop 
decision  rules  to  avoid  intransitive  results.  Generally  he 
found  the  authors  had  to  restrict  the  characteristics  of  the 
individual  rank  orders,  such  as  not  allowing  tied  rauiks 
(Dobbins,  1980a: 20). 

An  example  of  one  of  these  restricted  inputs  is  Bowman 
and  Colantoni’s  report.  If  the  majority  rule  problem  has 
transitivity  constraints  on  the  input  voting,  then  it  can  be 
solved  with  integer  programming  models  (Bowman  and  Colantoni, 
1973) . 
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Introducti qn 

Besides  the  ease  of  application  of  the  AHP,  this  thesis 
chose  to  examine  the  AHP  because  it  was  a  more  a^jpropriate 
tool  to  provide  discrini nation.  One  of  the  goals  of  the 
Battel le  study  was  to  discriminate  between  alternatives. 
Although  MAUT  also  discriminates  well,  it  is  not  as  easily 
applied  in  survey  form,  nor  does  MAUT  allow  for  intransitive 
rankings.  The  AHP  incorporates  intransitive  rankings  in  its 
measure  of  consistency  which  can  be  used  as  a  figure  of  merit 
for  the  different  aggregation  methods.  Finally,  this  thesis 
saw  the  need  for  research  on  the  effect  different  aggregation 
methods  have  on  the  AHP  results. 

PciQci gles 

The  analytical  hierarchy  process  uses  pairwise 
comparisons  between  elements  of  the  same  level  in  a  decision 
hierarchy  to  arrive  at  a  priority  vector  for  these  elements 
<Saaty,  198Zs351.  Saaty  describes  the  three  principles  of  the 
AHP: 

1)  Hierarchic  representation  and  decomposition 
-  breaking  the  problem  down  into  separate  elements. 

2)  Priority  setting  -  ranking  the  elements  by 
relative  importance. 

3)  Logical  Consistency  -  insuring  that 
elements  are  grouped  logically  and  ranked 
consistently  according  to  a  logical  criterion. 

(Saaty,  1982:26) 


Th«  slMiants  of  ths  hi  or  archy  mrm  clustered  into 
homogeneous  groups  of  five  to  nine  so  they  can  be 
meaningfully  compared  to  elements  in  the  next  higher  level. 
The  only  restriction  on  the  hierarchy  is  that  any  element  in 
the  next  higher  level  must  serve  as  a  criterion  for  assessing 
the  relative  impact  of  elements  in  the  level  below  (Saaty, 
1982:36).  A  hierarchy  can  be  divided  into  sub-nier archies 
sharing  only  a  common  top  most  element.  This  is  seen  in  this 
thesis  hierarchy.  Figure  1.  The  clusters  of  subob jecti ves  for 
the  third  level  are  not  related  between  clusters  except 
through  the  top  goal:  "TAV  Effectiveness."  Also,  the 
interrelationship  within  clusters  is  limited  to  the  "parent" 
objective  in  the  level  above  them. 

Kamenetzky  breaks  the  AHP  into  two  phases.  The  first 
phase  of  the  AHP  is  to  determine  the  relative  importance  of 
the  objectives.  Every  objective  is  assumed  to  have  an  unknown 
measure  of  relative  importance  denoted  Mi ,  i  «  l,...,p.  If 
these  weights  Wi ,  are  known  and  a  matrix  C  with  elements  Ci j 
=  Wi/Wj  is  constructed,  this  matrix  satisfies  the  equation: 

CM  =  pM 

Where:  p  =  order  of  the  matrix 

C  -  matrix  of  relative  importance 
U  -  (Wl,  W2,  ...,  Wp) 

(Kamenetzky,  1982:700) 

The  decision  maker  is  asked  to  provide  an  estimate  of 
the  matrix  C”  by  answering  the  following  question:  "What  is 
the  relative  importance  of  objective  i  with  respect  to 


objective  j?"  Saaty  has  shOMn  (1982)  that  an  estimate,  W’ , 
of  the  true  vector  of  weights  U  satisfies  the  equations 

C’W’  »  (lambda  maK)W’ 

where: Clambda  max  is  the  maximum  eigenvalue  of  C' 1 

Weighting  or  setting  the  priorities  requires  the 
objectives  in  each  level  be  compared  among  themselves  in 
relation  to  the  next  objective  "up  the  chain.”  These  in  turn 
are  prioritized  with  respect  to  the  objective  of  the  next 
higher  level,  and  so  on.  Finally,  each  of  these  priority 
vectors  is  multiplied  by  the  "parent”  element  in  the  next 
higher  level  to  obtain  overall  priorities.  This  is  done  by 
coming  down  the  hierarchy  and  weighting  the  priorities 
measured  in  a  level  with  respect  to  an  objective  in  the  next 
higher  level  with  the  weight  of  that  objective. 

The  second  phase  of  the  AHP  that  Kamenetzky  describes  is 
to  determine  the  relative  standing  of  each  alternative  with 
respect  to  each  objective.  There  may  be  as  many  as  p  pairwise 
comparison  matrices  (one  for  each  objective)  for  each  level 
of  the  hierarchy.  The  overall  score  of  each  alternative  is 
then  determined  by  simply  summing  the  weighted  valuations  of 
the  alternatives. 

One  of  the  weaknesses  of  AHP  is  that  it  has  few  axioms 
for  rigorous  derivations  of  its  validitv.  Saatv  admits  to  not 
making  a  “serious  attempt"  to  axiomatize  the  AHP,  but  lists 
the  following  assumptions  that  must  be  made  prior  to  applying 
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1.  A  systm  can  ba  daconposod  into  comparabla 
cl  asMs. 

2.  Elements  in  each  class  may  be  conpared  with 
respect  to  some  or  all  the  elements  in  an  adjacent 
class. 

3.  Comparisons  can  ba  made  in  terms  of  an 
absolute  numerical  scale  to  form  ratios. 

4.  The  pairwise  comparisons  utilize  reciprocal 
matrices  (optional). 

5.  Intrauisi ti vity  is  allowed  and  its  effect  on 
the  consistency  of  the  outcome  measured. 

6.  The  priority  of  an  element  is  derived 
through  a  weighting  principle. 

7.  Any  element  which  appears  in  the  hierarchy 
is  considered  relevant,  although  its  priority  may 
be  low.  It  does  not  make  sense  to  speak  of 
“irrelevant  alternatives"  introduced  into  the 
hierarchy  to  test  for  independence  from  them. 

8.  The  expressed  preferences  are  assumed  to  be 
deterministic  rather  than  probabal istic.  Thus  a 
preference  remains  fixed  and  is  not  contingent  upon 
other  factors  not  included  in  the  problem. 

(Saaty,  1980:9,248) 


The  assumptions  that  concern  this  thesis  are  2  and  8: 
individuals  caui  make  comparisons  using  the  ratio  scale  and 
the  elements  are  deterministic. 

Scale^  The  inputs  to  the  AHP  can  be  considered  a  type 
of  "score"  for  that  particular  objective.  Kraweic  lists  the 
following  advantages  of  scoring  methods: 

-  They  are  the  only  methods  specifically  designed 
to  incorporate  noneconomic  criteria. 

-  They  use  input  data  in  the  form  of  subjective 
estimates  provided  by  knowledgeable  persons  in  the 
form  of  point  or  interval  statistical  estimates. 

-  They  use  subjective  "guesses"  overtly,  where 
other  methods  generally  require  a  more  costly  and 
quantified  form  of  the  same  guess. 

-  The  subjective  probability  assessment  can  be  the 
conceptual  and  analytical  framework  of  the  scoring 
method  to  produce  an  efficient  portfolio  of  R&D 
projects. 

-  The  scoring  methods  produce  results  that  are,  on 
an  average.  90%  rank  orde^  consistent  with  economic 
and  constrained  models. 

(Kraweic,  1984:22) 
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TABLE  II 

AHP  Pairwise  Canparison  Scale 


INTENSITY 

IMPORTANCE 

OF 

DEFINITION 

EXPLANATION 

1 

Equal  importance  in 
both  elements. 

Two  elements  contribute 
equally  to  the  property 

3 

Weak  importance  of  one 
el ement  over  another 

Experience  and  judgement 
slightly  favor  one 
element  over  another 

5 

Strong  importance  of 
one  element  over 
another 

Experience  and  judgement 
strongly  favor  one 
element  over  another 

7 

Demonstrated  importance 
of  one  element 

An  element’s  dominance 
is  demonstrated  in 
practi ce 

9 

Absolute  importance  of 
one  element  over  another 

The  evidence  favoring 
one  element  over  another 
is  of  the  highest 
order  of  affirmation 

2,  4,  6,  8 

Intermediate  values  be¬ 
tween  two  judgements 

Compromise  is  needed 
between  two  judgements 

Reciprocals:  If  activity  i  has  one  of  the  preceding  numbers 

assigned  to  it  when  compared  with  activity  j, 
then  j  has  the  reciprocal  value  when  compared 
to  1.  (Saaty.  1982:78) 

The  scoring  function  in  the  AHP  has  a  unique  scale  shown 
in  Table  II.  The  -9  to  -*-9  ratio  scale  used  in  the  pairwise 
comparisons  is  based  on  Saaty’ s  tests  to  maximize  man’s 
ability  to  discriminate  between  alternatives.  Judges  range 
their  comparisons  based  from  absolutely  more  important  (9), 
through  equally  important  (1).  to  absolutely  less  important 


(-9).  Tha  scala,  however,  is  surrounded  with  the  controversy 
between  rat io*-intarval -ordinal  scales  and  the  requirements  to 
establish  these  measures  in  value  theory  (see  Value  Theory  in 
Chapter  2) . 

The  AHP  scoring  function  and  hierarchic  representation 
enables  decision  makers  to  represent  the  simultaneous 
interaction  of  many  factors  in  complex,  unstructured 
situations.  It  helps  them  to  identify  and  set  priorities  on 
the  basis  of  their  objectives  and  their  knowledge  and 
experience  of  each  problem.  Saaty  suggests  that  one's 
feelings  and  intuitive  judgments  are  probably  more 
representative  of  our  thinking  and  behavior  than  are  our 
verbalizations  of  them  (Saaty,  1982:12).  Perhaps  this  is  the 
reason  Gear  found  that  managers  readily  accepted  the  AHP 
(Gear,  pl8) . 

Saaty  lists  the  following  advantages  of  the  AHP: 


Unity:  Provides  a  single,  easily  understood, 
flexible  model  for  a  wide  range  of  unstructured 
problems. 

Process  Repetition:  Enables  people  to  refine 
their  definition  of  a  problem  and  to  improve  their 
judgment  and  understanding  through  repetition. 

Complexity:  integrates  deductive  and  systems 
approaches  in  solving  complex  problems. 

Judgment  and  Consensus:  Does  not  insist  on 
consensus  but  synthesizes  a  representative  outcome 
from  diverse  judgments. 

Interdependence:  Can  deal  with  the  inter — 
dependence  of  elements  in  a  system  and  does  not 
insist  on  linear  thinking. 

Tradeoffs:  Takes  into  consideration  the 
relative  priorities  of  factors  in  a  system  and 
enables  people  to  select  the  best  alternative  based 
on  their  goals. 

Hierarchic  Structuring:  The  AHP  reflects  the 
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natural  tendency  of  the  mind  to  sort  elements  of  a 
system  into  different  levels  and  to  group  like 
el ements  i n  each  1 evel . 

Synthesis:  Leads  to  an  overall  estimate  of  the 
desirability  of  each  alternative. 

tleasurement :  Provides  a  scale  for  measuring 
intangibles  and  a  method  for  establishing 
priorities. 

Consistency:  Tracks  the  logical  consistency  of 
judgments  used  in  determining  priorities. 

(Saaty,  1982:23) 

Consistency 

iQ^CoductigOj^  The  last  advantage  Saaty  lists  is 
consistency.  The  AHP  measures  the  "consistency”  in  the 
judgements.  Humans  are  seldom  perfectly  consistent  in  making 
comparative  judgements,  particularly  when  dealing  with 
intangibles  where  there  are  no  scales  to  use.  However , 
consistency  is  an  important  criterion  as  a  necessary 
condition  to  valid  scalings  of  reality  (Saaty,  1980:228) 
Rather  than  creating  axioms  requiring  only  consistent 
comparisons,  the  AHP  accepts  inconsistency  and  measures  the 
amount.  This  ratio  consistency  is  best  illustrated  with  an 
example.  If  car  A  costs  3  times  more  than  car  B,  and  that  car 
B  costs  2  times  more  than  car  C,  to  be  perfectly  consistent 
the  decision  maker  must  say  that  car  A  costs  6  times  more 
than  car  C.  Note  that  the  consistency  will  drop  if  he  says 
car  A  costs  5  times  more  than  car  C,  even  though  his  ordinal 
ranking  remains  the  same. 

Sroups  also  exhibit  inconsistency.  Several  authors  show 
that  groups  in  general  cannot  make  decisions  in  the  same 
rational  straight  forward  manner  that  an  individual  can.  One 
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May  to  look  at  group  decisions  is  to  compare  it  to  a  pairwise 
comparison  matrix.  Each  row  would  represent  one  member’s 
inputs,  whether  they  ave  ordinal  ranks  or  direct  assessment 
ratings.  There  would  inevitably  be  inconsistencies  between 
rows  (judges)  in  the  group.  This  would  be  an  indication  of 
the  group  inconsistency. 

Group  decisions  are  neither  rational  nor  straight 
forward  according  to  group  decision  properties  that  Arrow 
postulated  in  1951,  as  examined  in  the  literature  review 
(Blin  and  Satterthwai te,  1978:257).  The  crux  of  Arrow’s 
impossibility  theorem  is  that  intransitivity  is  generic  to 
group  decisions  that  comply  with  his  logical  conditions. 

The  participants  will  have  more  confidence  in  the  final 
decision  if  they  see  their  inputs  reflected  in  the  group 
output  vector.  Because  intransitve  ranks  between  judges  tend 
to  cancel  each  other,  the  fewer  the  “inconsistencies,"  the 
more  likely  the  participants  will  be  satisfied  with  the  group 
priority  vector.  From  this  approach,  one  may  consider  the 
minimum  group  intransitivity  as  a  valid  objective  in 
aggregating  individual  opinions. 

AHPi,  The  AHP  derives  a  consistency  index  by  taking  the 
ratio:  (lambda  max  -  n)  /  (n-1),  where  lambda  max  is  the 
trace  of  the  comparison  matrix  and  n  is  the  number  of 
criteria.  Saaty  explains  the  reasoning  behind  Cl  in  the 
following  manner.  If  the  comparisons  are  estimated  precisely, 
then  Xi j  -  Mi /W j  and  each  main  diagonal  element  would  be 
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divided  into  two  sections!  a  single  page  Battel le 
questionnaire  (Delphi  iteration),  and  a  16  page  instrument 
applying  the  AHP. 

The  "Delphi”  applicatic3n  included  statistical  group 
-feedback  (mean  and  standard  deviation)  -from  Battel le's  first 
quest ionai re  along  with  the  particular  respondent’s  original 
rating.  He  was  asked  if  he  wished  to  update  his  ratings  based 
on  his  current  knowledge  (feedback  experience).  In 
particular,  he  was  asked  to  explain  his  ratings  that  remained 
outside  one  standard  deviation  of  the  initial  group  average. 
The  Delphi  approach  sometimes  requires  several  iterations  to 
reduce  the  variance  between  respondents.  Dal  key  showed  that 
the  variance  between  respondents  has  the  most  dramatic 
reduction  after  the  second  Delphi  iteration  (Dal key, 

1970:10),  so  this  effort  will  be  limited  to  two  iterations  of 
the  survey. 

The  second  section  of  the  first  survey  solicited  132 
pairwise  comparisons  for  the  criteria  in  the  revised  Battel le 
study.  The  judges  were  also  asked  to  rate  their  own 
qualifications  associated  with  the  comparisons.  The  purpose 
was  to  have  a  basis  to  weight  the  opinions  of  the  more 
knowledgeable  judges.  Dalkey  found  these  self  ratings 
statistically  improved  the  accuracy  of  group  estimates  of 
historical  facts  (Dalkey,  1969c:2). 

The  second  survey  (Appendix  B)  returned  the  judges’ 
pairwise  matrices  and  the  corresponding  priority  vectors  so 
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ProcssSfi.  process  to  achieve  these  objectives  will 

be  two-fold.  First,  the  initial  Battel le  direct  assessent 
survey  will  be  returned  to  the  judges  with  the  group  results 
to  allow  the  judges  to  "update”  their  original  inputs.  The 
returned  AHP  survey  will  also  have  the  explanations  that 
individuals  gave  on  the  initial  survey  for  exceeding  one 
standard  deviation  of  the  group  mean.  The  final  collected 
surveys  will  be  used  for  the  determination  of  the  priority 
vectors. 

The  second  part  of  the  process  is  to  calculate  the  AHP 
priority  vectors  and  the  majority  rule  (MR)  priority  vector. 
Kendall’s  tau  will  measure  the  correlation  between  the 
resultant  priority  vectors.  The  Kendall’s  coefficient  of 
consistence  (zeta)  will  be  calculated  for  each  of  the  MR 
matrices  and  will  be  compared  to  Cl  (AHP  consistency  index). 

Finally,  as  an  ancillary  thesis  effort,  these  new 
priority  vector  will  be  applied  to  the  original  Battel le 
study  to  see  the  effect  that  the  AHP  has  on  the  outcome.  That 
is,  the  weights  will  be  applied  to  the  attribute  values  used 
by  Battel le  to  achieve  a  scaler  measure  of  the  TAV 
"feasibility"  for  each  of  the  original  proposals. 

Survey 

Appendix  A  shows  the  18  page  pamphlet  that  was  sent  to 
each  of  the  original  13  members  of  the  TAV  steering 
committee.  The  cover  page  explained  the  purpose  of  the  studv 
and  explained  how  to  complete  the  survey.  The  survey  was 
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ObjectiyeSf.  This  thesis  Mill  apply  the  AHP  to  solicit 
weights  throughout  the  1983  “Batelle"  study  of  the  TAV 
feasibility.  There  are  two  main  objectivesr  I)  Measure  the 
effect  that  feedback  has  on  the  rank  of  the  top  level  weights 
between  the  AHP  and  direct  assessment  techniques,  and  ID 
Measure  the  effect  that  different  aggregation  techniques  have 
on  the  priority  vectors  and  consistency  measures. 

Aiternati ye_Aggrgaches£.  The  Delphi  approach  is  an 
iterative  loop  to  achieve  consensus  through  controlled 
feedback.  The  Delphi  approach  (or  versions)  should  be  the 
first  step  in  all  aggregation  techniques  applied  to  group 
decision  making  through  surveys.  The  three  mathematical 
approaches  emphasized  in  this  study  of  the  AHP  are:  1)  taking 
the  geometric  mean  of  the  judge’s  inputs  before  calculating 
the  eigenvector,  2)  taking  the  arithmetic  mean  of  the  12 
judges’  eigenvectors,  and  3)  converting  the  pairwise 
comparisons  to  a  voting  matrix  and  applying  majority  rule. 

MeasureB_oi_Ef f ectiyenesSi  The  effect  of  feedback  will 
be  evaluated  by  the  variance  between  elements  of  the  judges’ 
priori  tv  vectors  (criteria  weights),  before  and  after 
feedback.  The  effect  of  different  aggregation  rules  will  be 
evaluated  by  the  change  in  consistency. 
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conparison  batwaan  tha  dif-farant  mathoda.  Tha  najority  rula 
vector  Mill  ba  referred  to  as  VoteM,  for  vote  mean. 

Aggraqation  Sumwary.  The  vector  aiaan  is  tha  most 
widely  used  technique  of  all  those  available  (Cook  and 
Seiford,  1978:1722).  This  average  vector,  when  it  is  derived 
from  a  voting  matrix,  is  also  the  method  recommended  by 
Kendall  to  minimize  the  mean  square  error  (Kendall, 

1955:100).  The  question  is  to  determine  the  relationship 
between  these  vectors  in  terms  of  their  consistency  and  their 
rank  correlation. 
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Note  that  the  arithmetic  mean  input  is  inappropriate 
with  ratio  inputs.  This  is  clear  with  an  example.  1-f  one  Nere 
to  take  the  simple  arithmetic  mean,  he  would  arrive  at  (a 
l/a)/2,  rather  than  one,  which  is  the  correct  ratio  mean 
between  two  opposite  ratings. 

Besides  the  priority  vector  derived  from  the  average 
input,  this  study  looked  at  the  priority  vector  derived  -from 
the  average  output.  The  arithmetic  mean  priority  vector  was 
determined  between  judges,  rather  than  the  geometric  mean 
priority  vector.  The  reason  was  that  the  arithmetic  mean 
preserves  the  restriction  that  the  sum  o-f  the  vector  elements 
equal  one.  The  geometric  mean  vector  would  require  an 
additional  normalization  before  it  would  satisfy  this 
restriction.  A  superficial  examination  showed  that  the 
normalized  geometric  mean  vector  and  the  arithmetic  mean 
vector  were  related,  but  the  extent  of  this  relationship  was 
not  examined  in  this  study. 

naigr^ty_Rul.e£.  The  final  aggregation  technique 
examined  will  be  to  convert  the  AHP  inputs  into  ordinal 
"votes"  and  sum  across  the  rows  of  the  comparison  matrix. 
These  row  totals  will  be  summed  across  judges  to  arrive  at 
the  Shannon  Majority  Rule  priority  rankings  as  explained  in 
the  literature  review.  The  interval  between  the  resulting 
ranks  will  be  the  difference  in  the  number  of  votes.  This 
conversion  to  an  interval  scale  priority  vector  will  allow 
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consistency  index  (Cl).  The  AHP  consistency  index  provides  a 
convenient  measure  a-f  the  consistency  of  pairwise 
preferences,  but  it  does  not  indicate  the  social  intricacies 
of  group  interactions,  such  as  dominant  individuals.  Saaty 
suggests  that  one  of  the  most  promising  contributions  of  the 
hierarchical  analysis  is  to  structure  the  problem  from  the 
start  with  interacting  groups  (Saaty,  1980:69). 

Aggregation  Methods.  The  point  of  the  aggregation 
method  is  to  accurately  reflect  the  true  “group”  priority 
vector.  This  thesis  will  look  at  two  of  the  many  methods  for 
aggregating  individual  opinions  into  one  opinion:  1)  taking 
the  mean  of  the  inputs/outputs  for  the  AHP,  and  2)  converting 
the  inputs  into  an  ordinal  scale  and  using  majority  rule.  In 
addition  two  techniques  to  reduce  the  variance  and 
intransitivity  between  priority  vectors  will  be  examined: 
emphasizing  inputs  of  certain  individuals  and  using  Delphi  to 
elicit  the  weights. 

t!§*Qs.  The  inputs  to  the  AHP  matrix  are  ratios 
between  the  values  of  two  objectives.  The  geometric  mean  is 
the  preferred  input  rating  when  determining  the  group  average 
input.  This  is  the  same  as  the  arithmetic  mean  of  the  input 
logarithms.  That  is,  the  nth  root  of  the  product  of  n  inputs 
is  equal  to  the  sum  of  n  input  logarithms  divided  by  n.  For 
example,  if  one  person  assigns  the  value  a  and  another 
assigns  the  value  1/a,  the  mean  would  be: 

C  ( 1/a)  (a)  lexp  ( 1 /2)  =  anti  logC  (log  (1/a)  log(a))/21 

1  =  1 
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ftoaraqation 

Introduction.  Saaty  described  the  AHP  as  providing  a 
new  method  to  incorporate  judgements  o-f  several  people  and 
resolve  conflicts  among  them  (Saaty.  1982:25).  The  drawback 
is  how  to  accomplish  this  amalgamation: 

How  to  represent  group  judgement  in  a  satisfactory 
way  when  people’s  experiences  and  judgments  differ, 
and  whose  opinions  should  be  taken  more  seriously 
and  why,  is  a  major  problem  in  social  study  and 
conflict  analysis. 

(Saaty,  1980:69) 

Unfortunately,  this  “major  problem"  does  not  have  an 
easy  solution  with  the  AHP.  Saaty  defines  consensus  as 
improving  confidence  in  the  priority  vector  by  using  several 
judges  to  bring  the  results  in  line  with  majority  preferences 
(Saaty,  1980:66).  For  large  matrices,  trying  to  achieve  a 
consensus  via  survey  becomes  intractable  because  of  the 
enormous  amount  of  feedback  required  for  the  n(n-l)/2 
comparisons.  The  more  direct  approach  would  be  to  seek 
consensus  for  the  derived  priority  vectors.  The  judges  would 
then  reexamine  their  comparisons  in  light  of  the  feedback  for 
the  n-element  priority  vector. 

Saaty  assumes  that  the  input  comparisons  to  the  AHP  are 
either  individual  inputs  or  the  aggregated  group  inputs. 
However,  there  has  been  no  study  examining  the  effect  that 
different  aggregation  methods  have  on  the  results  of  the  AHP. 
The  AHP  results  include  the  priority  vector  and  the 


zeta  when  the  sane  matrix  is  converted  to  ordinal 
comparisons.  This  is  because  the  ratio  may  change  between 
equivalent  comparisons  but  the  ordinal  relationship  remains 
the  saune.  This  research  will  examine  the  relationship  between 
Cl  avid  zeta  as  a  function  of  the  size  of  the  matrix  and  the 
judge  making  the  comparisons  to  more  fully  understand  the 
interaction  between  them. 
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at  n  s  8,  tha  calculatad  zata  must  be  lower  than  .60  to 
reject  that  zeta**!  (Dobbins,  1980as  32-35) .  Another  way  to 
look  at  it  is  that  the  nore  objectives  the  judge  is  voting 
on,  the  more  circular  triads  he  can  have  and  still  be 
considered  consistent. 

The  null  hypothesis  will  be  that  there  is  perfect 
disagreement  between  the  judges  (there  is  no  concordance 
between  methods).  Kendall  developed  methods  and  special  small 
N  value  (#  judges)  probability  tables  to  evaluate  W  between 
zero  and  one  (Appendix  E) .  Kendall  does  not  explain  how  he 
calculated  the  distribution  of  d  (or  zeta) ,  other  than  to  say 
that  the  "distributions  are  rather  troublesome  to  obtain" 
(Kendall,  1970:147). 

§y0!DdC^«.  the  single  judge,  pairwise  comparisons 

provide  an  additional  measure  of  consistency  that  is  not 
available  in  direct  assessment  (interval  scale)  or  ranking 
(ordinal  scale).  The  consistency  measure  for  the  continuous 
scale  is  the  AHP  consistency  index  (Cl)  and  for  the  ordinal 
scale  it  is  Kendall’s  zeta. 

Cl  can  be  thought  of  as  the  amount  of  divergence  each 
comparison  is  from  the  equivalent  "average"  comparison 
computed  from  the  eigen  vector.  Zeta  can  be  thought  of  as  the 
count  of  the  number  of  circular  triads  over  the  maximum 
number  of  circular  triads  possible. 

It  is  believed  that  a  matrix  with  ratio  scale  (AHP)  mav 
show  inconsistency  with  Cl  but  remain  transitive  according  to 
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and  tha  ranking  is  nora  inconsistant.  Tha  actual  formulas  for 


zeta's  calculation  are  discussad  in  the  next  chapter, 

Methodol ogy . 

Note  that  it  is  possible  to  have  circular  polyads 
greater  than  three.  The  circular  three  ranks  (triad),  however 
is  always  present  in  a  larger  “n-ad.”  Kendall  points  out  that 
a  circular  n-ad  must  contain  at  least  n-2  circular 
triads (Kendall,  1970:145). 

The  fact  that  it  contains  circular  triads  does  not  imply 
that  it  is  itself  circular.  For  example,  suppose  that  ABCD  is 
circular.  Then  either  A>C  or  C>A.  In  the  first  case  ACD  is 
circular,  in  the  second  case  ABC  is  circular.  Similarly 
either  ABD  or  BCD  is  circular.  Thus  the  tetrad  must  contain 
at  least  two  circular  triads.  Kendall  ignored  the  more 
ambiguous  criteria  based  on  larger  polyads  and  based  zeta  on 
circular  triads  which  comprise  the  elementary  inconsistencies 
(Kendall,  1970:146). 

Kendall  insists  that  the  data  can  be  ranked  only  if  zeta 
=  1,  i.e.,  there  are  no  triads  (Kendall,  1970:147).  Kendall 
determines  statistically  when  zeta=l  by  establishing  a 
distribution  zeta  would  have  if  all  the  preferences  were 
allotted  at  random.  The  distribution  is  dependent  on  n.  For  a 
given  significance  level,  the  calculated  zeta  value  that 
rejects  the  null  hypothesis  that  zeta^l  slowly  drops 
(Appendix  E).  That  is.  at  -05  significance,  when  n  =  3,  any 
calculated  zeta  different  from  1  will  reject  zeta=l;  whereas 
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scale  (1  to  9  scale  for  the  AHP)  or  by  an  ordinal  scale 
(-lyOyl  or  O, 1/2, 1)  such  as  in  ranking.  This  thought  is  nore 
clearly  brought  out  by  Klahr. 

Klahr  'found  that  information  about  transitivity  is  lost 
in  rank  ordering  (single  judge).  He  thought  paired 
ccsmparisons  were  better  than  ranking  in  the  sense  that  equal 
preference  in  ranking  may  be  the  result  of  intransitivities 
in  pairwise  comparisons.  A  distinction  that  may  be  importaint 
(Klahr,  1969s 319).  That  is,  if  A>B>C  in  one  line  of 
comparisons  a^d  B>A>C  in  another  line,  then  the  end  result 
will  be  A— B>C.  Ranking,  on  the  other  hand  only  has  one  line; 
it  is  a  single  dimensional  vector  which  assumes  perfect 
transitivity  for  the  individual  judge. 

Vgti ng^Matri Xj.  The  index  which  determines  the  amount 
of  intransi ti vi ty  in  the  ordinal  pairwise  comparison  matrix, 
or  "voting  matrix,"  is  called  Kendall’s  coefficient  of 
consistence,  or  "Zeta."  Three  ranks  would  be  intransitive 
(inconsistent)  if  it  circled  back  on  itself:  A>B>C>A. 

Kendall  calls  an  intransitive  order  of  3  ranks  a  circular 
triad,  and  it  forms  the  basic  unit  in  his  coefficient  of 
consistence,  zeta.  Zeta  first  calculates  the  number  of 
"circular  triads"  (d)  among  n  ordered  criteria  (Kendall, 
1955:147).  The  ratio  of  d  to  the  maximum  number  of  triads  is 
subtracted  from  one  to  form  zeta.  That  is,  if  d=12  and  the 
max  number  of  triads  was  equal  to  20,  zeta  would  be  equal  to: 
1-  12/20  =  0.4.  If  there  are  more  circular  triads,  zeta  drops 
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dacision  makar  is  4 oread  to  coapara  altarnatxves  which  ara 
assoc iatad  with  axtrama  points  in  tha  consaquanca  spaca 
(Kaaanatzky,  1982:711). 

Ellsburg  notad  that  inconsistency  is  to  ba  expected, 
particularly  with  respect  to  utility  differences  that  are 
alnost  equal.  The  most  important  information  gained,  he 
states,  “concerns  choices  which  the  subject  finds  easy  to 
make"  (Ellsburg,  1954:274).  Saaty  tries  to  keep  the 
comparisons  in  a  given  matrix  (cluster  on  the  sane  level  of 
the  hierarchy)  within  a  factor  of  10  of  each  other,  to  avoid 
the  extreme  points.  If  one  of  the  factors  is  outside  this 
range  it  should  be  removed  and  placed  in  another  matrix 
comparing  more  similar  factors,  otherwise,  the  AHP 
comparisons  are  distorted  and  meaningless. 

Saaty  cautions  users  not  to  take  "short  cuts"  with  the 
AHP,  such  as  asking  subjects  to  fill  the  cells  of  only  one 
row/column  and  then  computing  the  other  cells  from  these. 

This  short  cut  assumes  the  judge  will  be  perfectly  consistent 
in  completing  the  remainder  of  the  matrix.  In  this  case  the 
eigenvalue  will  be  equal  to  n  which  represents  perfect 
consistency  (Saaty,  1980:228). 

Consistency  can  be  measured  for  the  single  judge  when  he 
has  a  number  of  equivalent  judgements  that  can  be  compared. 
Pairwise  comparisons  present  different  rows  in  the  matrix 
that  represent  equivalent  comparisons,  which  can  be  related. 
It  does  not  matter  if  the  preferences  are  measured  by  a  ratio 
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consistency  needed  to  maintain  coherence  among  the  objects  O'# 
one’s  experiences  and  the  inconsistency  necessary  for  human 
intellectual  growth  (Saaty,  1982:82).  If  the  CR  is 
considerably  more  than  .10,  say  .20,  then  a  reexamination  of 
the  judgments  may  be  in  order  (Saaty,  1982:9-1). 

It  is  important  to  emphasize  that  the  objective  is  to 
make  “good"  decisions,  not  to  minimize  the  CR.  The  intensity 
of  concern  with  consistency  and  inconsistency  differs  by  an 
order  of  magnitude.  That  is,  if  a  unit  amount  of  priorities 
were  divided  among  the  two,  consistency  would  be  about  .90 
and  inconsistency  about  .10.  Good  decisions  are  most  often 
based  on  consistent  judgments  but  the  converse  is  not 
necessarily  true.  It  is  very  passible  to  make  perfectly 
consistent  judgments  that  are  wrong. 

To  measure  the  inconsistency  of  all  the  judgments  made 
in  the  hierarchy,  the  Cl  for  each  cluster  is  multiplied  by 
the  priority  of  the  criterion  with  respect  to  which  these 
comparisons  are  made.  This  number  is  summed  over  all  the 
criteria  which  gives  a  single  overall  weighted  number. 

Kamenetzky  notes  that  this  consistency  index  measure  may 
be  high  even  though  the  judgements  are  consistent  because  the 
decision  maker’s  value  function  may  not  be  linear.  This 
should  not  happen  if  the  attributes  are  truly  mutually 
preferentially  independent.  It  is  not  unusual,  however,  for 
attributes  that  seem  mutually  preferentially  independent  on 
the  first  evaluation,  to  turn  out  to  be  not  so  when  the 
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equal  to  1.  Therefore,  the  trace,  or  sum  of  the  main  diagonal 
entries,  tuould  be  equal  to  n,  Mhere  n  is  the  dimension  of  the 
perfectly  consistent  matrix.  The  trace  Mill  always  be  equal 
to,  or  greater  than  the  dimension  of  the  matrix.  Saaty  calls 
this  a  “monotone  increasing  function"  of  Xi j  (Saaty, 

1974:57).  Since  the  trace  is  equal  to  the  maximum  eigen 
value,  or  lambda  max,  one  may  look  at  the  difference  (lambda 
max  -  n)  as  a  measure  of  the  average  ratio  deviation. 

The  AHP  consistency  index  can  be  thought  of  as  a 
measurement  of  the  average  amount  each  paired  comparison 
deviates  from  computed  ratio  (Ui/Wj  from  the  eigenvector). 
Gear  describes  Saaty' s  consistency  index  as  a  measure  of  the 
"fuzziness"  introduced  by  the  answers  to  pairwise  ratio 
comparison  scales  (Gear,  pi 3). 

When  the  Cl  is  zero  there  is  complete  consistency.  To 
decide  how  acceptable  this  number  was,  Saaty  determined  what 
the  average  Cl  was  from  a  randomly  generated  comparison 
matrix  of  different  dimensions  and  called  it  the  random  index 
(Rl).  Using  a  sample  size  of  500  he  obtained  the  following 
Rls: 

Random  Indexes 

n:  3  4  5  6  7  8 

ci(ri):  .58  .90  1.12  1.24  1.32  1.41 

(Saaty,  1980:21) 

The  ratio  of  Cl  to  Rl  for  the  same  order  matrix  is 
called  the  consistency  ratio  (CR) .  Saaty  allows  some 
inconsistency  in  order  to  provide  a  balance  between  the 
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the  judges  could  verify  that  the  derived  vectors  reflected 
their  “true"  opinions.  The  priority  vectors  wcN'e  drawn  as 
histograns  to  ease  the  interpretation.  Their  original  inputs 
were  displayed  to  give  them  the  opportunity  to  correct 
potential  errors  in  their  pairwise  comparisons. 

To  assist  the  judges  in  identifying  possible  errors 
(reversed  signs,  etc),  the  comparisons  that  contributed  the 
most  to  the  inconsistency  of  the  matrix  were  circled.  The 
judge  was  then  asked  to  verify  that  the  circled  comparisons 
were  what  he  had  intended.  This  was  accomplished  only  for 
those  matrices  that  exceeded  10%  CR. 

These  comparisons  were  isolated  because  they  had  the 
largest  ratio  of  pairwise  ratings  (Xij)  divided  by  the  ratio 
of  the  priority  vector  elements  (Wi/Wj).  This  is  the  method 
recommended  by  Saaty  as  being  the  most  relevant  (Saaty, 
l'?80;66).  The  comparisons  were  isolated  serially.  That  is, 
after  the  first  "faulty"  input  was  corrected  with  the  nearest 
integer  rating,  the  eigenvector  was  recalculated.  If  the  CR 
remained  above  iOX,  the  comparison  that  contributed  the  most 
to  the  inconsistency  of  the  revised  matrix  was  isolated,  and 
so  on  -  until  the  CR  dropped  to  lOX.  Appen  *'  x  Cl  shows  the 
computer  program  to  identify  the  potential  errors.  As  seen  in 
Appendix  B,  the  judge  was  not  given  anv  reason  why  the  inputs 
were  circled  nor  the  order  of  their  importance,  to  prevent 
biasing  the  results.  He  was  simply  asked  to  verify  that  the 
circled  rating  was  what  he  had  intended. 


Statistical  -feedback  was  considered  inappropriate  -for 
the  AHP  portion  because  it  would  require  an  excessively 
"busy"  survey.  It  would  entail  displaying  both  the  group  mean 
vectors  and  the  judge’s  own  vectors  for  nine  matrices. 
Instead,  a  type  of  Press’s  Quality  Feedback  Control  (QFC)  was 
used  to  improve  the  consensus  of  the  PHP  judgments. 

The  feedback,  shown  in  Appendix  B,  consisted  of  three 
pages  of  anonymous  reasons  why  individuals  had  deviated 
outside  one  standard  deviation  of  the  group  mean  for  the  top 
level  criteria.  Although  only  the  top  level  was  originally 
solicited,  many  of  the  judges  included  reasons  for  their 
ratings  for  subsequent  criteria.  All  the  written  responses 
were  included  in  the  second  survey.  Appendix  B. 


Analysis 

iQtCQdyctigQf.  As  explained  earlier,  the  purpose  a4 
this  analysis  was  to  examine  the  e-ffect  of  applying  three 
different  aggregation  methods  on  both  the  consistency  and 
final  priority  vectors  using  the  AHP  and  to  discover  the 
critical  parameters.  The  actual  application  of  the  AHP  and 
majority  rule  was  accomplished  using  three  pascal  programs 
written  on  an  Apple  lie  computer.  Appendix  C.  Saaty's  basic 
interactive  code  was  converted  into  pascal  to  interact  with 
files  (Saaty,  1982:167).  The  files  contained  each  decision 
maker’s  ratings  for  the  entire  hierarchy  (Sample  file  in 
Appendi x  0) . 

These  programs  provided  the  tool  to  accomplish  the  two 
major  objectives:  1)  measuring  the  effect  of  feedback,  and  2) 
measuring  the  effect  of  the  aggregation  technique.  The  two 
statistical  measures  used  to  evaluate  the  significance  of  the 
correlation  between  the  derived  priority  vectors  were 
Kendall’s  tau  for  two  vectors,  and  Kendall’s  concordance,  W, 
for  n-vectors. 

The  Kendall  rank  correlation 

coefficient  is  suitable  as  a  measure  of  correlation  with  data 
that  IS  at  least  ordinal.  The  tau  will  give  a  measure  of  the 
degree  o<  association  or  correlation  between  the  two  sets  of 
ranks.  The  sampling  distribution  of  tau  under  the  null 
hvpothesis  is  known,  and  therefore  tau,  unlike  Spearman’s  rho 


(another  correlation  coe-fficient  between  ranks),  is  subject 
to  tests  of  significance. 

One  may  think  of  tau  as  a  function  of  the  mini  muni  number 
of  inversions  or  interchanges  between  neighbors  which  is 
required  to  transform  one  ranking  into  another.  That  is,  tau 
is  a  measure  of  the  disarray  of  a  series  of  pairwise 
comparisons  (Siegel,  1956:316).  It  is  determined  by 
calculating  the  value  of  the  "disarray”  (S)  by  starting  with 
the  first  number  on  the  left  and  counting  the  number  of  ranks 
to  its  right  which  are  larger.  Then  the  number  of  ranks  to 
its  right  which  are  smaller  are  subtracted.  This  number  is 
determined  for  all  the  ranks  and  summed  to  obtain  S. 

It’s  calculation  is  best  shown  in  an  example.  Suppose 
judge  Y  and  X  ranked  the  objectives  a,b,c  and  d  in  the 
fallowing  order: 

□b jecti ves 
abed 

Judge  X  3421 

Judge  Y  3142 

Rearranging  the  order,  so  that  judge  X’s  appear  in  natural 
order: 

d  c  a  b 

Judge  X  1234 

Judge  Y  2431 

Judges  X’s  revised  rankings  are  in  their  natural  order, 
so  their  contribution  to  S  is  0.  One  next  proceeds  to  Y  to 
determine  how  many  pairs  of  ranks  in  judge  Y’s  set  are  in 
their  correct  (natural)  order  with  respect  to  each  other. 


First  look  at  all  possible  pairs  oi  ranks  in  which  Y’s  rank 
2,  the  rank  farthest  to  the  left  in  the  set,  is  one  raember  of 
the  pair.  The  first  pair,  (2&4),  has  the  correct  order;  2 
precedes  4,  so  a  score  of  -t-i  is  assigned  to  this  pair.  The 
second  pair  of  ranks  (2&3)  is  also  in  the  correct  order  so  it 
also  earns  a  score  of  -fl.  The  third  pair  (2&1}  are  not  in 
natural  order  so  this  pair  has  a  score  of  -1.  The  total  score 
for  all  pairs  which  include  the  rank  2  is: 

(+1)  +  (  +  1)  +  (-1)  =  +1. 

The  pairs  of  ranks  which  include  the  second  from  the 
left  number  (4)  are:  (4&3)  and  (4Stl),  which  get  assigned  (-1) 
and  (-1)  respectively  for  a  score  of  -2.  When  looking  at  rank 
3  and  Svjcceeding  ranks,  there  is  only  one  pair  (38cl)  which  is 
scared  as  -  1.  The  total  of  all  the  scores  is  then: 

(+1)  +  (-2)  +  (-1)  *  -2  «  S 

Tau  is  then  determined  by  dividing  this  number  (S)  by  the 
maximum  amount  of  "disarray"  possible,  which  is  N(N-l)/2. 

This  would  have  been  yielded  if  the  rankings  of  judges  X  and 
Y  had  agreed  perfectly,  because  every  pair  of  judge  Y’s  ranks 
would  be  in  the  same  (correct)  order  as  judge  X,  and  thus 
every  pair  would  receive  a  score  of  -(-I.  The  maximum  possible 
total  would  occur  in  the  case  of  perfect  agreement  between  X 
and  Y,  which  would  be  six  (four  things  taken  two  at  a  time). 

Tau  is  the  degree  of  relation  between  the  two  sets  of 
ranks  and  is  indicated  by  the  ratio  of  the  actual  total  of 
+l’s  and  -I’s  to  the  possible  maximum  total: 
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tau  a  actual /possi bl e  total  »  -2/6  =  -.33 
The  tau  »  -.33  is  a  measure  04  the  agreement  betMeen  the 
ranks  assigned  to  the  objectives  by  judge  X  and  those 
assig.ied  by  judge  Y. 

When  tNo  or  more  observations  on  either  the  X  or  the  Y 
variable  are  tied,  the  tied  values  are  given  the  average  o-f 
the  ranks  they  would  have  received  i4  there  were  no  ties.  The 
effect  of  ties  is  to  change  the  denominator  for  tau.  In  the 
case  of  ties,  tau  becomes: 


tau  =  CN(N-l)/2  -  Txr^N(N-l)/2  -  Tyj''^ 

where:  Tx  =^t.lt^l)  and  t^is  the  number  of  tied 

i  observations  in  each  group i 

Ty  =  same  as  Tx  for  group  y 

Example: 


Judge  X  (Eigen  VM> : 

Judge  Y  (vote  mean): 

rearranging  on  X: 

Judge  X: 

Judge  Y: 

calculating  S  i 
S  =  (+5)  +  (+4)  +  ( 


Objectives 
a  b  c  d  e  f 

6  5  3  1  2  4 

5.5  5.5  3.5  1  2  3.5 


dec  f  b  a 
1  2  3  4  5  6 

1  2  3.5  3.5  5.5  5.5 

the  usual  way: 

+  (2)  +  (O)  =  13 


There  are  two  sets  of  tied  ranks.  Two  objectives  are 

tied  at  rank  3.5  and  two  are  tied  at  5.5  (both  by  "Judge  Y"). 

In  each  of  these  cases,  t=2,  the  number  of  tied  observations 

.  Thus  Ty  may  be  computed: 

Ty  =it,(trl)/2 

=  C2(2-l)  +  2(2-1) 1/2 
=  2 

tau  =  13  =  13  =  .931 

C15-0l'^*  C  15-21’'^  13.96 
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This  sx ample  shows  actual  data  and  is  a  correlation 
between  the  eigenvector  means  (EigenV/M)  and  the  majority  rule 
mean  (Votell)  shown  in  Table  111. 

Kendall  developed  a  table  to  evaluate  the  significance 
of  tau  for  N  <  10.  This  is  copied  in  Appendix  E.  For  N  >  8, 
the  sampling  distribution  of  tau  is  practically 
indistinguishable  from  the  normal  distribution  (Siegel, 
1956:220).  Appendix  E  may  be  used  to  determine  the  exact 
probability  associated  with  the  occurrence  (one-tailed)  under 
the  null  hypothesis  (vectors  are  correlated)  of  any  value  as 
extreme  as  an  observed  S. 

The  sampling  distributions  of  S  and  tau  are  identical, 
in  a  probability  sense.  In  as  much  as  tau  is  a  function  of  S, 
either  might  be  tabled,  but  Kendall  found  it  more  convenient 
to  tabulate  S.  For  such  small  samples,  the  significance  of  an 
observed  relation  between  two  samples  of  ranks  may  be 
determined  by  simply  finding  the  value  of  5  and  then 
referring  to  Appendix  E  to  determine  the  one-tailea 
probability  associated  with  that  value.  If  the  P  <  alpha 
(probability  of  rejecting  a  true  hypothesis),  the  null 
hypothesis  may  be  rejected.  In  the  above  example,  entering 
Appendix  E  with  5=13  and  N=6,  one  reads  .0083  as  the 
probability  of  associated  with  values  as  large  as  13. 
Arbitrarily  picking  alpha  =  .05,  one  rejects  that  the  vectors 
are  not  correlated  and  accepts  the  alternate  hypothesis  that 
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the  vectors  are  correlated. 

Kendall . Concordance  _M.  To  examine  the  effect  of 
feedback,  this  thesis  examined  the  overall  agreement  between 
ranks  derived  by  the  Delphi  process,  the  AHP,  and  majority 
rule  voting.  to  evaluate  if  one  method  resulted  in  more 
"consensus"  than  the  other.  Also  this  evaluation  was 
necessary  to  evaluate  the  effect  of  feedback  for  the  Delphi 
process. 

As  a  solution  to  the  problem  of  ascertaining  the  overall 
agreement  among  N  sets  of  rankings,  it  might  seem  reasonable 
to  find  all  the  tau’s  between  all  possible  pairs  of  the 
rankings  and  then  compute  the  average  of  these  coefficients 
to  determine  the  overall  association.  Unless  N  is  very  small, 
such  a  procedure  would  be  extremely  tedious.  The  variance 
between  judges  for  each  abjective  would  reflect  this 
agreement.  However  there  is  no  statistical  test  for  the 
significance  of  this  measure.  This  thesis  chose  to  apply  the 
Kendall  Coefficient  of  Concordance,  which  is  a  function  of 
the  variance.  The  tables  to  test  the  significance  of  W  are  in 
Appendix  E. 

The  Kendall  Coefficient  of  Concordance  (W)  was  used  to 
statistically  evaluate  the  independence  of  the  N  judges’ 
rankings.  The  12  judges’  priority  vectors  for  the  top  level 
objectives  were  converted  to  ranks  for  the  eigenvector  and 
direct  assessment  approaches.  The  concordance  test  evaluated 
if  the  clusters  of  vectors  for  each  approach  were  independent 
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of  each  other 


There  are  sax  steps  in  the  determination  of  Ws 


1.  Let  k  -  the  number  of  entities  to  be  ranked,  and 
let  N  =  the  number  of  judges  assigning  ranks.  Cast 
the  observed  ranks  in  a  k  X  N  table. 

2.  For  each  objective,  determine  Rj,  the  sum  of  the 
ranks  assigned  to  that  objective  by  the  N  judges. 

3.  Determine  the  mean  of  the  Rj.  Express  each  Rj  as 
a  deviation  from  that  mean.  Square  these 
deviations,  and  sum  the  squares  to  obtain  5. 

4.  If  the  proportion  of  ties  in  the  N  sets  of  ranks 
is  large,  add  .n  the  T  factor  below  (This  tie 
correction  is  similar  Kendall’s  correlation  for  two 
vectors;  tau) . 

_ S _ 

W  =  2  3 

N  (k  -  k)/12  -  N^T 

where 

3 

T  =  y  it  -  fc)/i2 

<■  k 

I. 

where: 

t  =  number  of  observations  zn  a  group  tied  for 

a  given  rank  and  is  summed  over  all  group 
of  ties  within  any  one  of  the  N  rankings. 

5.  The  method  for  determining  whether  the  observed 
value  of  w  is  significantly  different  from  zero 
depends  on  the  size  of  k: 

a.  If  k  is  7  or  smaller.  Appendix  E  gives 
critical  values  of  S  associated  with  W’ s 
significance  at  the  .05  and  .01  levels. 

b.  If  k  is  larger  than  7,  the  chi  square 
distribution  ==  N(k-l)W  whose  significance,  for 
df  =  k~l,  may  be  tested  using  chi  square 
tables. 

(Siegel,  1956:237) 


The  following  shows  an  example  of  the  procedures  in 
calculating  the  concordance  between  the  first  five  judges’ 
rankings  of  the  top  level  objectives  (Appendix  F  shows  the 
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orginal  eigenvectors).  The  ranks  are  shown  belows 

Judges’  Rauiks  for  the  Top  Level  Objectives 


Judge 

1  2  3  4  5  Rj 


Requirements 

Performance 

Costs 

Risk 

Logistics 
Sur vi vabi 1 i ty 


6  4  4.5 

3  5  4.5 

4  3  6 

113 

5  2  2 

2  6  1 


6  4  24.5 

5  5  22.5 

1  6  18.0 

4  1  10.0 

2  2  13.0 

3  3  15.0 


The  mean  of  the  Rj  is  17.17.  The  sum  of  the  squared 
deviations  from  this  mean  is  S  =  53.8  28.4  .7  -<■  51.3  * 

17.3  +  4.7  =  156.3.  With  k=6  and  N=5,  S  must  be  at  least 
182.4  to  be  significant  at  the  .05  level.  Therefore,  one 
cannot  reject  the  null  hypothesis  that  the  first  five  judges’ 
rankings  are  unrelated  to  each  ether.  The  conclusion  is  that 
the  agreement  among  the  5  judges  is  no  higher  than  it  would 
be  by  chance.  The  complete  results  will  be  discussed  in  the 
results  chapter. 

Although  the  proportion  of  ties  is  small,  for  the  sake 
of  this  example,  the  correction  factor  T  will  be  calculated. 
Judge  #3  has  one  set  of  ticr^s  requirements  and  performance  is 
tied  at  4.5.  Therefore,  the  number  of  tied  objectives  (2)  are 
summed  over  the  number  of  ties  per  judge  (1): 


T( judge  1-2)  =  O 

2 

T(judge  #3)  =  tit  -1)/12  =  2(4-l)/12  =  .5 

T ( judge  4-5)  =  O 
The  sum  of  T  j  =  0.5 

s _ ^  156.  a... 

W=  k2(N’-  N)/12  -  k^T  i6)(5-5)/12  -.5  =  359.5=  .435 
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The  effect  of  ties,  if  left  uncorrected,  is  to  slightly 
depress  the  value  of  W  and  make  the  decision  more 
conservative.  That  is,  one  is  slightly  less  likely  to  reject 
the  null  hypothesis  if  he  had  ties  and  did  not  correct  for 
them.  To  determine  the  significance  of  the  revised  W  one  must 
convert  it  in  terms  of  S  because  Appendix  E  is  designed 
around  s.  The  conversion  is  acggmpl ished  by  multiplying  the 
original  5  (1S6.3)  by  the  ratio  of  the  old  denominator  (360) 
to  the  corrected  denominator  (3S9.5).  This  corrected  S  is 
then  used  to  determine  the  significance  at  k^h  and  N=5. 

A  high  or  significant  value  of  W  may  be  interpreted  as 
meaning  that  the  observers  or  judges  are  applying  essentially 
the  same  standard  in  ranking  the  N  objects  under  study.  Often 
their  pooled  ordering  may  serve  as  a  "standard,"  especially 
when  there  is  no  relevant  external  criterion  for  ordering  the 
objects  (Siegel,  1956:237). 

Feedback_Ef fecti  Before  evaluating  different 
aggregation  techniques,  it  is  important  to  discover  the 
effect  that  feedback  has  on  AHP  and  compare  it  to  the  effect 
feedback  has  on  the  direct  assessment  technique  (Battelle). 
The  variance  among  the  12  judges*'  criteria  weights  (top  level 
vector)  derived  by  AHP  before  and  after  feedback  will  be 
compared  with  the  corresponding  variance  change  for  the 
Battelle  vectors.  This  variance  will  isolate  those  criteria 
on  which  there  was  little  aggreement.  The  agreement  in  the 
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overall  rankings  of  the  vectors  will  be  measured  by  the 
Kendall  Coefficient  of  Concordance,  “U. "  Kendall's  W  will 
show  if  there  is  agreement  amongst  the  judges  for  each 
method:  direct  assessment  and  the  AHP  and  it  will  discover  if 
the  feedback  significantly  changed  the  rank  order  of  either 
method. 

Aggregati.gn_Ef f ecti  A  separate  issue  from  feedback  is 
how  to  most  accurately  aggregate  the  opinions  of  the  group 
into  one  group  opinion.  This  study  will  examine  three 
mathematical  methods  to  arrive  at  the  "group  opinion:"  1) 
calculate  the  arithmetic  mean  of  the  eigenvectors  of  the  12 
judges,  2)  calculate  the  geometric  mean  of  each  pairwise 
comparison  for  the  12  judges  prior  to  computing  an 
eigenvector,  and  3)  convert  each  judge's  matrix  into  a  voting 
matrix  and  use  majority  rule  to  arrive  at  a  weighted  vector. 
The  remainder  of  the  chapter  discusses  each  of  these  methods 
in  more  detail. 

Ari thmeti c_Meani  The  arithmetic  mean  of  the  weights  is 
the  most  straight  forward  of  the  four  methods.  Each  judge’s 
criterion  weight  (an  element  of  the  eigenvector)  will  be 
summed  across  the  12  judges  and  divided  by  12.  The  mean  Cl 
will  be  the  average  of  the  12  judges'  matrix  Cls.  Note  that 
this  thesis  did  not  examine  the  geometric  mean  of  the  12 
judges'  eigenvectors.  This  has  been  left  for  further 
research. 

Seoinetri.c_lneut_Meani  Each  judge’s  criterion  rating 


Mill  be  multiplied  across  the  12  judges  and  the  12th  root 
will  be  used  as  the  input  rating.  Because  the  inputs  are 
averaged  prior  to  the  calculation  of  an  eigenvector,  the 
resulting  consistency  index  will  reflect  the  “true"  Cl, 
versus  the  nwaan  arithmetic  Cl  calculated  above. 

f1aj.ority_Rul.ea.  Each  matrix  will  be  converted  to  a 
“voting"  matrix  by  converting  the  AHP  ratio  inputs  into 
"votes."  Each  AHP  rating  greater  than  one  will  be  assigned 
one  (vote)  and  each  rating  less  than  one  will  be  assigned 
zero.  Each  AHP  rating  equal  to  one  will  be  assigned  one  half 
of  a  vote.  The  priority  vector  will  be  the  percentage  of 
total  votes  received  by  each  criterion.  The  voting  program 
(Appendix  C4)  shows  the  pascal  code  to  convert  the  AHP  matrix 
into  a  voting  matrix. 

ZetH:.  The  consistency  measure  for  the  majority  rule 
matrices  will  be  Kendall's  Coefficient  of  Consistence  (zeta). 
Zeta  measures  the  degree  of  intransitivity  among  n  ordinal ly 
ranked  elements  determined  by  n(n-l)/2  paired  comparisons. 

In  preference  matrices  of  more  than  three  alternatives, 
it  is  possible  to  have  the  majority  preferences  of  three 
alternatives  aligned  to  be  circular  triads  (Dobbins, 
1980a:67).  For  example,  an  aggregate  may  have  a  rank  order  of 
A=C  >  B=E=F  >  D  which  has  five  circular  triads:  ACDA,  ABDA, 
AEDA,  AFDA,  and  BEFB.  Kendall  proved  that  the  maximum  and 
minimum  number  of  triads  can  be  attained  by  arrangement  of 
preferences.  Kendall's  equation  for  d,  the  number  of  circular 
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riads  in  a  preference  matrix,  consisted  of  the  number  of 
liter  natives  and  the  sum  of  the  votes  in  row  i  (Ai)  of  the 


ireference  matrix: 

d  =  ntn-)(n-2)/6  -  cX  Ai  (Ai-1>  1/2 

L 

[eta  is  1  -  d/(max  number  of  triads)  where  the  maximum  number 
]f  triads  depends  whether  n  is  odd  or  even: 

odd  even 

3  3 

max  triads  =  (n  -n)/24  (n  -4n)/24 


The  derivation  of  zeta  is  based  on  Ai  being  integer 
valued  (no  tied  pairs).  To  resolve  this  problem  the  model  for 
this  research  brackets  the  possible  values  if  fractional 
pairs  of  Ai’s  exist.  The  steps  of  the  d  bracketing  are  in  the 
computer  procedure  "CalcZeta"  in  Appendix  C4  and  are 
descrioed  below  (Dobbins,  1980a:29): 

1)  Arrange  Ai  row  totals  in  order  of  their  value. 

2)  Count  the  number  of  fractional  Ai  row  totals. 

3)  Round  the  upper  one-half  of  each  of  the  fractional  Ai 
values  upward  to  their  next  larger  integer  values. 

4)  Round  the  lower  one— half  of  each  fractional  Ai  values 
downward  to  their  next  smaller  integer. 

5)  Verify  that  the  sum  of  the  rounded  Ai’s  =  n(n-l)/2. 

6)  Use  this  rounded  set  of  Ai’s  to  calculate  a  lower  d 
(gives  a  "low"  zeta  even  though  it  is  a  high  d  estimate) 

7)  Return  to  the  ordered  unrounded  Ai’s  and  round  the 
upper  one-half  of  each  pair  of  the  fractional  Ai  values 
downward  to  their  next  smaller  integer  values. 

8)  Round  the  lower  one-half  of  each  pair  of  the 
fractional  Ai’s  upward  to  their  next  larger  integer 
values. 

9)  Verifv  that  the  sum  of  the  second  rounded  Ai’s  = 
n (n-1) /2. 

10)  Calculate  an  upper  d  based  on  this  second  rounded 
set  of  Ai  ’  s. 

11)  Average  the  lower  d  and  upper  d  to  form  an 
approximate  d  for  the  matrix  with  the  tied  pairs. 
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The  corrected  ze'ta  above  (zeta(c))  will  always  show 
improvement  over  the  uncorrected  zeta  (zeta(u))  when 
f ractionized  Ai’s  exist.  Therefore,  there  is  less 
discrimination  with  zeta(c)  than  with  zeta(u)  and  any 
reference  to  zeta  will  imply  zeta(u).  When  the  distinction  is 
necessary,  it  will  be  made  clear  to  the  reader. 

It  can  readily  be  seen  that  zeta^^l  when  there  are  no 
circular  triads,  which  is  the  only  time  that  the  data  can  be 
ranked.  (Kendall,  1970:147).  Kendall  determined  the 
distribution  zeta  would  have  if  all  the  preferences  were 
allotted  at  random  (Appendix  E) .  The  distribution  is 
dependent  on  n  for  a  given  significance  level.  As  n 
increases,  the  calculated  zeta  value  that  rejects  the  null 
hypothesis  (zetasi)  slowly  drops.  That  is,  at  .05 
significance,  when  n  =  3,  any  calculated  zeta  other  than  one 
will  reject  zeta-1;  whereas  at  n  =  8,  the  calculated  zeta 
must  be  lower  than  .60  to  reject  that  zeta=l  (Dobbins, 

1980a: 32-35) . 

Another  way  to  look  at  zeta's  significance  i s  to  compare 
it  to  the  AHP  Cl.  Kendall  changes  his  zeta  limit  for  a  given 
significance  depending  on  n,  but  the  source  zeta  remains  the 
same.  Saaty,  on  the  other  hand,  changes  the  source  Cl 
depending  on  n  (Cl  to  CR) ,  but  not  the  cut  off  (.10).  These 
relative  cutoffs  do  not  concern  this  research  because  the 
focus  here  is  on  zeta  and  Cl  as  comparative  measures  of 
effectiveness.  These  comparisons  are  between  matrices  of  the 
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same  dimensions  (n>,  'and  so  the  cutO'^f  is  irrelevant. 

Summary 

This  thesis  will  determine  the  variance  reduction 
brought  about  by  feedback  of  group  information  for  both 
direct  assessment  and  the  AHP.  It  will  also  compare  the 
aggregated  priority  vectors,  both  in  rank  order  and  the  level 
of  consistency.  There  will  be  three  types  of  aggregated 
vectors:  1)  The  eigenvector  calculated  from  the  geometric 
mean  of  individual  comparisons  (AHP),  2)  The  average  vector 
calculated  from  12  judges'  eigenvectors  (AHP),  and  3)  The 
vector  calculated  using  the  majority  rule  process. 


QblBt€!I-.V£._Resul  ts 

Introduction 

The  analysis  -focussed  on  three  aspects  of  aggregating 
the  results  gained  by  the  AHP.  The  first  aspect  was  the 
effect  feedback  had  on  increasing  the  agreement  between  the 
judges.  The  Kendall  Coefficient  of  Concordance  (W)  was  used 
to  statistically  evaluate  this  agreement.  Several  other 
aspects  of  feedback  were  also  examined. 

The  second  aspect  was  an  analysis  of  the  aggregation 
method:  the  AHP  geometric  mean  input,  the  AHP  vector  mean, 
and  majority  rule  vote  mean. The  effect  on  the  consistency 
measure  and  the  resulting  rank  orders  was  examined. 

The  third  aspect  was  a  closer  examination  of  the 
consistency  measures:  Cl  and  zeta.  The  equality  of  their 
means  was  determined  by  analysis  of  variance  varying  the 
judge,  the  matrix  size,  and  whether  the  consistency  measure 
was  Cl  or  zeta.  The  Kendall  Correlation  Coefficient,  tau. 
indicated  the  extent  the  rank  orders  were  the  same  between 
consistency  measures. 

Finally,  an  ancillary  effort  was  to  apply  the  final 
priority  vectors  (“bottom  vector")  to  the  original  Battel le 
attribute  values  and  examine  the  difference  the  AHP  would 
make  to  the  original  study.  Battel le  found  the  results  were 
insensitive  to  large  changes  in  the  weights,  but  they  did  not 
go  beyond  the  top  level.  This  thesis  sought  to  discover  if 


Battel le’s  results  Nould  change  if  the  eeights  mere  elicited 
throughout  the  hierarchy. 

Feedback 

The  effect  of  feedback  Mas  evaluated  by  the  change  in 
Kendall’s  concordance  (M)  between  the  original  Battel le 
survey  (OelphiO)  and  the  repeat  Battelle  survey  (Delphi  1). 

The  original  Battelle  survey  elicited  direct  assessment 
ratings  between  1  and  10  for  the  top  level  weights  only.  This 
thesis  returned  the  same  questions  on  the  Battelle  survey 
along  with  the  group  mean  rating  and  standard  deviation  for 
each  objective  in  the  top  level.  Each  judge  was  asked  to 
reevaluate  his  original  ratings  (accomplished  one  year 
earlier)  and  to  explain  any  of  his  new  ratings  that  remained 
outside  of  one  standard  deviation  of  the  original  mean. 

Twelve  of  the  original  13  judges  on  the  TAO  steering 
committee  who  responded  to  the  original  Battelle  survey  were 
located.  Eleven  completed  the  survey  and  justified  their 
ratings  for  the  top  level  weights.  An  additional  judge  from 
the  TAV  program  office  completed  the  AHP  portions  (judge  #5), 
but  did  not  participate  in  the  Delphi  portion  of  the  study. 

The  AHP  portion  of  the  survey  was  returned  with  "Quality 
Feedbacks"  an  additional  three  pages  of  reasons  that  the 
judges  gave  to  justify  their  ratings  of  the  top  level,  and 
often  subsequent  levels  of  the  hierarchy  (Appendix  B) .  Each 
page  of  the  repeat  AHP  survey  displayed  the  resulting 
priority  histogram  and  the  original  responses  of  each 
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recipient’s  comparison  matrix.  He  was  asked  to  rate  (1-10) 
hoM  well  the  histogram  reflected  his  "true"  priorities.  If  he 
rated  the  histogram  poor,  he  was  asked  to  review  his  original 
AHP  inputs. 

Judges  were  asked  to  review  certain  compatrisons  of  the 
original  inputs  to  insure  they  were  correct.  These 
potentially  "erroneous"  comparisons  were  located  by  computer 
as  the  ones  that  contributed  the  most  to  an  excess  of  a 
consistency  ratio  equal  to  .10.  They  were  circled  on  the 
returned  matrix. 

QlIebLiL  This  thesis  examined  the  overall  agreement 
between  ranks  derived  by  the  Delphi  process  to  evaluate  the 
effect  of  feedback  for  direct  solicitation  of  weights.  The 
ratings  are  shown  for  both  Delphi O  (Original  Battel le>  and 
Delphil  in  Table  III.  Table  Ilia  shows  the  weights  converted 
to  ranks,  and  Table  Illb  shows  the  statistical  correlations. 
The  effect  of  feedback  was  measured  by  Kendall’s  concordance, 
W,  and  the  variance  of  the  judge’s  priorities. 

As  seen  in  Table  lllb,  feedback  reduced  the  variance  and 
increased  the  concordance  (agreement)  between  the  judges.  The 
significant  U  (sig.  <  .01)  shows  that  there  is  statistically 
significant  agreement  between  both  Del  phi O  and  Delphil 
priority  ranks.  The  significant  Tau  shows  the  two  average 
vectors  are  also  correlated.  Feedback  did  not  affect  the  rank 
order  of  the  priority  vector.  This  can  be  seen  graphically  in 
the  histogram  in  Figure  3. 
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TABLE  III 


Top  Level  Battel le  Ratings 


Battel le  Original  (Delphi 0) 


□bj 

1 

2 

3 

4 

6 

7 

8 

9 

19 

11 

12 

.  Ave 

1 

10 

10 

10 

10 

10 

10 

10 

7 

10 

10 

10 

9.7 

2 

6 

8 

7 

6 

6 

9 

5 

5 

10 

7 

8 

7.2 

3 

6 

8 

B 

10 

8 

8 

10 

9 

6 

8 

7 

8. 1 

4 

8 

9 

10 

10 

7 

5 

8 

9 

4 

6 

10 

7.8 

5 

6 

5 

10 

7 

5 

7 

6 

9 

6 

6 

8 

7.0 

6 

8 

5 

6 

7 

7 

6 

10 

8 

8 

7 

9 

7.5 

7 

8 

5 

9 

8 

8 

10 

8 

7 

7 

6 

4 

7.5 

8 

5 

1 

5 

5 

2 

5 

5 

4 

3 

3 

3.7 

( Battel le. 

1983 

:6-2) 

□bj 

Second 

Survey 

(Delphi  1 ) 

1 

2 

3 

4 

6 

7 

8 

9 

10 

11 

12 

Ave 

1 

10 

10 

10 

10 

10 

9 

10 

7 

10 

10 

10 

9.6 

2 

6 

S 

7 

5 

7 

7 

7 

7 

10 

7 

5 

6.6 

3 

7 

9 

8 

10 

8 

10 

10 

9 

7 

8 

8 

8.5 

4 

8 

9 

10 

10 

7 

5 

8 

9 

5 

6 

9 

7.8 

5 

6 

8 

10 

6 

6 

6 

5 

9 

6 

6 

8 

6.9 

6 

8 

5 

6 

8 

8 

7 

10 

8 

10 

8 

9 

7.9 

7 

7 

5 

9 

8 

9 

10 

7 

7 

8 

7 

5 

7.4 

B 

4 

1 

5 

5 

5 

3 

4 

4 

3 

2 

3 

3.5 

Where: 

Judge  ttS:  Added  after  i 
not  included 
□b jecti ves: 

1:  Requirements 

2 :  Ooa 1 s 

3;  Performance 
4:  Costs 


al  Battel le  survey  so  was 
Delphi  feedback  data. 

Technology  Risk 
Logi sti cs/Supportabi 1 i ty 
Survi vabi 1 i ty 
Environmental  Impact 


ni  ti 
for 

S: 

6: 

7: 

8: 
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TABLE  Ilia 


Del phi 0  and 

Del phi  1  1 

Ranks 

Battel le  Survey 

(Del phiO) 

JUDGE 

OBJ 

1 

2 

3 

4 

5 

6 

7 

a 

9 

lO 

11 

AVE 

1 

8 

8 

7 

7 

8 

7.5 

7 

3.5 

7.5 

a 

7.5 

8 

2 

3 

5.5 

3 

2 

3 

6 

1.5 

2 

7.5 

5.5 

4.5 

3 

3 

3 

5.5 

4 

7 

6.5 

5 

7 

7 

3.5 

7 

3 

7 

4 

6 

7 

7 

7 

4.5 

1.5 

4.5 

7 

2 

3 

7.5 

6 

5 

3 

3 

7 

3.5 

2 

4 

3 

7 

3  e  ^ 

3 

4.5 

2 

6 

6 

3 

5 

5 

6.5 

7.5 

4.5 

3.5 

5 

3 

2 

4.5 

7 

6 

3 

2 

3.5 

4.5 

3 

7 

5 

6 

5.5 

6 

4.5 

8 

1 

1 

1 

1 

1 

1.5 

1.5 

1 

1 

1 

1 

1 

Del  phi -Battel le  Survey 

(Delphi  1 ) 

JUDGE 

OBJ 

1 

2 

3 

4 

5 

6 

7 

a 

9 

10 

11 

AVE 

1 

8 

a 

7 

7 

8 

6 

7 

3 

7 

8 

8 

8 

2 

2.5 

•r 

3 

2 

3.5 

3.5 

3 

7 

4.5 

2.5 

2 

3 

4.5 

6.5 

4 

7 

5.5 

7.5 

7 

7 

4 

6.5 

4.5 

7 

4 

6.5 

6.5 

7 

7 

3.5 

2 

5 

7 

2 

2m  5 

6.5 

5 

5 

2.5 

5 

7 

3 

2 

3 

2 

7 

3 

2a  S 

4.5 

3 

6 

4.5 

3 

5 

4.5 

7 

7.5 

3.5 

3 

t5 

4.5 

2.5 

4 

7 

6.5 

3 

2 

4.5 

5.5 

4.5 

7 

5 

7 

6.5 

6.5 

6 

8 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

TABLE  II lb 


Concordance  and  Tau  Between  Del  phi  0  and  Delphi  1 

Del  phi O  Del  phi  1 

Kendall’s  W  .516  .564 

Variance  1.44  1.22 

Kendall’s  Tau  Between  AveDelphiO  and  AveDelphil  =  . 

( si g< .  01 ) 

Where: 

Variance  is  equal  to  the  average  variance  -for  the 
ratings  on  the  8  top  level  objectives. 


837 


a.  q:  •-<  a  IT 


FieURE  3 


Top  Level  Priority  Histogran) 
Del  phi  0,  Delphi  1,  and  Eigen  GM 


I 

T 

Y 


o 

1 

1 . 0"^ 

-  .95 - 

.9 

- .  89  -  - 

■So'-''  .  84  OOOP 

.8 

joe  .  81  /  •  80*  * 

-  -  82  ■ ~ 

•“tie  77fr>r*r»<» 

<«>.77/.76-j 

.7 

-j.72/72n- 

.6 

.5 

.4 

—  .37  — 

_  .39 _ : 

.3 

req  per-f  cost  risk  logist  survive 


where: 

Del  phi  0: 


Del  phi  1 : 


Ei genGh: 


Rank  is  derived  from 
iterations,  i . e. .  it 
Battel le  received  by 
Rank  is  derived  -from 
one  iteration,  i.e., 
vector  after  feedback 


the  Delphi  approach  with  zero 
is  the  priority  vector 
direct  assessment, 
the  del  phi  approach  with 
it  is  the  Battelle  prioritv 
(direct  assessment). 


Rank  is  derived  from  the  eigen  vector  from  the 
geometric  mean  of  the  12  judges'  inputs. 
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AHP.Feedbackj^  Nine  of  the  twelve  judges  returned  the 
second  AHP  survey.  These  nine  judges  had  56  pairwise 
comparisons  that  caused  a  'few  of  their  matrices  to  exceed  10% 
consistency  ratio.  This  was  SX  of  the  total  number  of 
comparisons  made.  Of  these  56  potential  "errors”  that  were 
called  to  their  attention,  the  judges  changed  46X  of  them 
(26).  These  changes  included  17  rating  changes  for  an  average 
of  3.0  units  of  change  and  9  sign  changes. 

The  nonresponding  judges’  (10,11,12)  inputs  were 
corrected  for  major  errors  only-  These  "major"  errors  were 
identified  if  they  were  the  most  "deviant"  comparisons  and  if 
a  change  made  a  substantial  improvement  in  the  consistency 
(20X).  The  effort  was  intended  to  make  the  resulting  matrices 
more  representative  of  the  nonrespondi ng  judges’  opinions.  It 
was  not  an  effort  to  improve  the  consistency.  Appendix  J 
shows  the  revised  priorities  calculated  for  both  sets  of 
revised  groups:  the  9  judge  revised  (natural)  group  and  the 
12  judge  revised  group  (three  "computer — revised"  judges).  The 
priorities  between  these  groups  caused  only  minor  consistency 
and  rank  changes.  Table  IV  shows  the  respective  priorities 
and  IVa  shows  the  priorities  converted  to  ranks  and  the 


statistical  analysis 


TABLE  IV 

Original  and  Revised  AHP  Priorities  (EigenGM) 


Original  Surve 


E-f -fecti  ve 

Reauire  1 

Per -for: 

Loais^ 

Survive 

1 

0.312 

0.262 

0.479 

0.234 

0.202 

0.  059 

O 

A. 

0.  241 

0. 170 

0.  186 

0.382 

0. 161 

0.  150 

3 

0.  1 10 

0.064 

0.335 

0.384 

0.638 

0.  156 

4 

0.091 

0.119 

0.  135 

5 

0.  127 

0.116 

0.079 

6 

0.  120 

0.045 

0.  137 

7 

0.  106 

0.284 

8 

0.117 

cr 

0.  033 

0.030 

0.003 

0.001 

0.  005 

0.019 

ci 

0.  027 

0.  006 

0.006 

0.001 

0.009 

0.015 

Propel  1  ants 

Maintainable  1 

Maneuver 

1 

0.  376 

0.  127 

0.457 

0.219 

0.265 

0.368 

•?; 

0.405 

0.333 

0.  176 

4 

0.274 

cr 

0.  014 

0.  003 

0.008 

ci 

0.023 

0.004 

0.013 

Second  Sur 


i-ffective  Require 

Per-f  orm 

Ri  sks 

Loqi st 

Survi  VI 

. 297  . 239 

.478 

.240 

.  224 

.065 

:.  .261  .121 

.  184 

.  402 

.  147 

.  148 

.113  . 058 

.338 

.358 

.630 

.  149 

. 094  . 156 

.  125 

;  .120  .148 

.  089 

1  .120  .035 

.  136 

,  090 

.289 

1  .  154 

i  .006  .022 

.003 

.  000 

.  000 

.  010 

r  .005  .016 

.  005 

.  000 

.  001 

.  008 

Topellants  Maintainable  Maneuver 

.369  .106 

.  457 

.196  .282 

.  3t»8 

. 435  . 363 

.  176 

TABLE  IVa 


Ranks  and  Correlations  Between  Priority  Vectors 
First  AHP,  Second  AHP,  and  Vote  Mean* 


E-f  Tecti  veness  Requirements 


AHPl 

AHP2 

VoteM 

AHPl 

AHP2 

VoteM 

□ 

1  6 

6 

5.5 

1 

8 

8  ■ 

8  ■ 

B 

2  5 

5 

5.5 

1 

7 

4 

5.5 

J 

3 

1.5 

1.5 

2 

E 

1 

4.5 

6 

5.5 

C 

3 

4-5 

6 

5.5 

T 

“T 

1.5 

1*5 

1 

I 

4.5 

3 

t; 

V 

4.5 

6 

5.5 

E 

TAU 

AHPl  - 

1.0 

.96 

AHPl  - 

.67 

.84 

SI  q:  — 

.02 

.03 

siq: 

.05 

.02 

AHF 

7  — 

— 

.91 

sig: 

— 

.01 

Survi vabi  1  i tv 

Mai ntai nabi 

1  i  ty 

AHPl 

AHF2 

VoteM 

AHPl 

AHP2 

VoteM 

0 

1  1.5 

1 

1.5 

1 - 

1 

^  1 

1 

B 

2  4.5 

4.  5 

4.5 

2.5 

2.  5 

T 

J 

3  4.5 

mm 

4 

E 

4  4.5 

B9 

C 

5  2 

m 

T 

6  4.5 

■ 

1 

7  7 

■9 

mBm 

H 

9IH 

V 

E 

TAU 

AHPl:  - 

1 . 0 

.  96 

- 

1 . 0 

SI  q: 

i— - 

.  02 

.03 

■VMM 

.08 

^9i9 

TAD  (3  Element  Priorities):  1.0 


Sig:  .12 

t  Vote  Mean  derived  ^rom  second  AHP 

Priorities  considered  tied  within  .03  of  each  other 
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The  a-ffect  of  "quality"  feedback  on  the  AHP  was  not 
measured  because  the  judges  changed  inputs,  not  in  response 
to  the  quality  feedback,  but  in  response  to  the  prompting  to 
review  certain  comparisons.  Only  two  of  the  nine  returned 
surveys  that  had  changes  that  were  not  prompted.  These 
changes  were  the  result  of  the  judges'  dissatisfaction  with 
the  histogram  of  the  orginal  eigenvector.  One  judge  did  not 
like  the  7:1  ratio  for  one  pair  of  the  objectives  reflected 
in  the  histogram,  so  he  changed  the  comparison  to  5:1. 

Another  judge  changed  a  comparison  from  2:1  to  1.2:1  after  he 
saw  the  histogram. 

Feedback,_Cgn(:l  usi  gns.  The  AHP  seems  resilient  to 
feedback.  Even  when  directed  to  reexamine  isolated  pairwise 
comparisons,  545C  of  the  returned  comparisons  were  not 
changed.  Individuals  expressed  general  satisfaction  with  the 
AHP  priorities.  The  few  minor  changes  made  had  little  impact 
on  the  rank  order  of  the  priorities. 

□ne  surprise  was  that  the  top  level  AHP  weights 
correlated  more  strongly  with  the  original  Battel le  study 
than  to  the  second  iteration  (Table  V  and  Figure  3).  The 
original  Battel le  study  was  accomplished  a  year  earlier  than 
the  AHP  and  Delphi  iteration. 

One  needs  to  exercise  caution  prior  to  saying  this 
supports  Saaty's  contention  that  pairwise  comparisons  are 
better  able  to  elicit  the  underlying  opinions.  The  reason  for 
the  large  shift  in  tau  is  that  the  direct  assessment  weights 
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Cg(|B*'lison_Battel  l.e_Resul  ts 


The  final  ancillary  effort  of  this  thesis  Mas  to  examine 
the  difference  that  applying  the  AHP  Mould  make  to  the 
outcome  of  the  Battel le  study.  The  original  Battel le  results 
are  "generic"  in  this  study  to  avoid  proprietary  information 
from  defense  contractors.  The  reader  Mho  has  an  official  need 
may  contact  ASD/XR. 

To  compare  the  Battelle  final  scores  and  the  AHP  final 
scores,  this  study  applied  the  original  Battelle  Meights  to 
the  revised  32  criterion  hierarchy  rather  than  the  original 
47  criteria  hierarchy.  The  minor  shifts  in  the  TAV  scores 
betMeen  Battel le's  original  hierarchy  and  the  revised 
hierarchy  Mere  mainly  due  to  the  "single"  stage  concepts 
losing  points  Mithout  the  environmental  impact  objective.  The 
largest  shift  Mas  a  drop  from  5th  to  8th  place  (TAV  #  13). 

The  AHP  priority  vector  applied  for  the  Battelle 
comparison  Mas  derived  from  the  geometric  mean  of  the  nine 
judges  Mho  revised  their  original  inputs.  As  shoMn  earlier  in 
this  chapter,  the  difference  Mi th  the  12  judge  "revised" 
priority  vector  was  negligible  so  there  was  no  significant 
impact  on  this  results. 

Table  XII  shows  the  scores  for  the  14  TAV  designs  bv 
applying  the  bottom  priority  vectors  to  the  Battelle 
attribute  scores.  Figures  7  and  8  graphically  protray  the 
scares.  Table  XIII  shows  the  correlation  between  the  scores 
listed  in  Table  XII.  Since  they  are  all  significantly 
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looking  at  the  five  matrices  in  this  thesis  that  all  have  the 
same  dimension  (3),  the  relationship  breaks  down  between  Cl 
and  variance.  Table  XI  shows  the  tau  for  Cl  and  the  variance 
between  judges  is  insignificant  (sig.=.804)  for  the  five 
matrices  that  have  three  objectives.  Therefore,  this  research 
has  no  conclusive  information  to  show  a  stroi.g  relationship 
between  consistency  and  variance  between  judges. 

TABLE  XI 

Correlation  Between  Cl  and  Deviation  for  Matrix  Size  3 

Cl  Deviation  t  10 
. 038  1 . 98 

,106  1.57 

.106  2.26 

.188  1.75 

.230  2.15 

tau  =  .1054  Significance  =  .804 

6lQSi.ti_on_i  n_Hi  er  ar chyi_  It  was  thought  that  the  general 
or  specific  nature  of  the  oDjectives  may  affect  the 
consistency.  It  was  believed  that  the  nature  of  the 
objectives  was  mainly  determined  if  they  were  near  the  bottom 
of  the  hierarchy  or  near  the  top.  It  was  originally  thought 
that  the  more  technical,  lower  level  objectives  would  lead  to 
less  consistency  from  the  high  level  decision  makers  used  in 
this  study.  However,  comparing  the  consistencies  of  the 
matrices  (size  =  3)  with  their  scattered  distribution,  shows 
no  evidence  that  either  the  variance  or  the  consistency  of 
the  judgements  was  affected  by  level  in  the  hierarchy. 
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TABLE  X 


Correlation  Between  Consistency 

and  Average 

Matrix 

— 

Standard 

De vi at i ons  ( Between 

Judges' 

Priority  Vector  Elements)  1 

Matrix  Dimension 

Voting 

Matrix 

AHP 

Matri x 

Zeta 

Dev  t  10 

Cl 

Dev  t  10 

8 

.79 

.54 

.22 

.52 

7 

.81 

.73 

.  16 

.96 

6 

.83 

1.0 

.  14 

.92 

4 

.88 

1.4 

.  12 

1.52 

3 

.87 

.73 

.07 

1.75 

Tau  (significance) 

.738  (.082) 

.8 

(.051) 

Before  interpreting  this  relationship,  however,  one  must 
consider  the  strong  relationship  between  the  size  of  the 
matrix  and  consistency.  The  strong  correlation  between 
consistency  and  judge  variance  may  be  due  to  the  correlation 
between  matrix  size  and  judge  variance.  Figure  6  shows  that 
the  larger  the  matrix,  the  greater  the  variance  between 
abjective  weights. 


FIGURE  6 
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Dimension  of  Matrix 

One  may  incorrectly  interpret  Figure  6  as  showing  that 
consistent  judges  have  less  variance,  and  both  consistency 
and  variance  are  a  function  of  matrix  size.  However,  when 
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This  is  not  to  say  that  the  dimension  of  the  matrix  is 
irrelevant  to  zeta.  Zeta  counts  the  number  of  circular  triads 
and  as  such,  it  shows  greater  consistency  with  the  smaller 
matrices.  This  is  reflected  in  Table  VII,  which  shows  a 
strong  correlation  between  the  matrix  dimension  and  the 
consi stency . 

Juda®!.  Another  factor  that  contributes  to  the  variance 
O'*  the  consistency  is  the  individual  judge.  The  individual 
judges  accounted  for  a  significant  (sig.=.073)  portion  of  the 
variance  of  both  consistency  measures  (Apppendix  H) .  The 
judge  influence  is  significant  because  his  inconsistent 
judgement  will  cross  the  “boundaries"  between  the  other 
factors:  Cl,  zeta,  and  the  matrix  size. 

Pr  1  or  i  ty_Vector_yar  iancgs.  After  examining  the  strong 
correlation  between  consistency  with  judges  and  matrix  size 
and  the  role  of  variance  and  matrix  size,  the  next  question 
emerges:  Does  the  consistency  of  the  individual  affect  the 
actual  variance  between  judges.  In  other  words,  does  a  group 
of  consistent  individuals  result  in  less  variance  between 
priority  vectors  than  a  group  of  inconsistent  individuals? 
Table  X  shows  the  strong  correlation  between  consistency  and 
variance  between  judges  (alpha=.05).  This  may  lead  one  to  say 
that  the  more  consistent  the  judges  are,  the  greater  the 
variance  between  the  judges. 


lOfa 
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must  be  conpared,  whether  one  is  examining  ratio  consistency 
or  ordinal  transitivity. 

Saaty  developed  a  table  and  an  adjustment  to  Cl  to 
account  for  the  fact  that  consistency  becomes  better  as  a 
function  of  increasing  matrix  size  (Saaty,  1980:62).  Saaty 
divided  Cl  by  an  adjustment  for  the  matrix  size,  called  the 
ratio  index,  to  arrive  at  the  consistency  ratio  (CR) .  Kendall 
adjusted  the  significance  of  zeta  so  that  as  the  number  of 
objectives  being  compared  decreased,  the  tolerances  for  zeta 
narrowed.  That  is,  zeta  had  to  be  closer  to  one  with  smaller 
matrices,  to  be  statistically  equal  to  one  (see  the 
statistical  tables  in  Appendix  E) . 

This  relationship  between  consistency  and  matrix  size  is 
shown  by  the  significance  of  the  proportion  of  variance 
accounted  for  by  the  size  of  the  matrix  (Appendix  H) .  For 
zeta  the  significance  is  .677  and  for  Cl  the  significance  is 
.054. 

The  reason  why  matrix  size  (n)  is  not  as  important  to 
zeta  is  that  zeta  is  based  on  an  n-cubed  factor  divided  by 
another  n-cubed  factor  to  calculate  the  number  of  circular 
triads  and  the  maximum  number  passible  (see  methodology, 
chapter  5).  The  AHP’s  Cl,  on  the  other  hand,  solves  an  nth 
order  equation  to  calculate  the  eigen  values.  Thus,  the  size 
of  the  matrix  has  a  larger  influence  in  the  determination  of 
Cl,  especially  for  the  larger  matrices,  than  it  does  in  the 
determination  of  zeta. 
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This  test  rejects  the  null  hypothesis  of  equal  means  at  a 
significance  =  .078.  One  must  accept  that  the  Cl  and  zeta  are 
not  the  same  in  magnitude.  This  is  a  reasonable  conclusion  as 
the  total  mean  Cl  =  .06  and  total  mean  zeta  =  .08. 

Although  they  are  not  statistically  the  same  size.  Cl 
and  zeta  are  highly  correlated  measures  of  consistency.  Table 
IX  shows  the  strong  (alpha=.001)  relationship  between  zeta 
and  Cl.  This  significance  is  stronger  than  originally 
expected  and  illustrates  that  Cl  reflects  the 
intransitivities  among  the  rankings  and  that  zeta  may  also 
reflect  the  "strength”  of  these  intransitivities. 


TABLE  IX 


Kendall’s  Correlation  Tau  for 

Consi stency 

Cl 

Matrix  Size 

Zeta  -.4518  (.001) 

.1520  (.006) 

Cl 

.1674  (.002) 

Before  too  strong  a  conclusion  can  be  drawn  about  the 
Cl-zeta  correlation,  one  must  examine  how  the  different 
factors  affect  them  both.  One  of  these  factors  is  the  matrix 
size. 

Matrix _Sizei.  The  dimension  of  the  comparison  matrix 
has  a  strong  influence  on  the  consistency  of  the  judgements, 
whether  that  measure  is  Cl  or  zeta.  The  consistency  drops  for 
both  Cl  and  Zeta  as  the  matrix  size  increases.  This  is 
expected  because  the  chances  for  finding  a  consistent 
framework  are  smaller  the  larger  the  number  of  objects  that 
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Cgnsisten£^ 

This  axami nation  sought  to  discover  the  similarity  of 
Saaty’s  consistency  index  (Cl)  and  Kendall’s  coefficient  of 
consistence  (zeta).  Their  correlation  with  each  other,  the 
number  of  objectives  being  compared,  and  the  judge  making  the 
comparison  were  determined  through  Kendall’s  tau  and  analysis 
of  variance  (anova). 

The  zeta  value  was  subtracted  from  one  to  make  it 
conceptually  equal  to  the  Cl.  That  is,  both  measures  ranged 
from  zero  for  perfect  consistency  to  one  for  random 
consistency.  This  would  have  no  effect  on  the  statistical 
evaluations  made. 

Qi_and_Jgta4.  The  126  pairs  of  consistency  measures  C9 
clusters  X  (12  judges  -f  geometric  mean  input  arithmetic 
mean  vector)!  were  divided  into  two  groups:  Cl  and  zeta 
measures.  Table  VI II  shows  the  summary  of  the  anova  results 
between  Cl  and  zeta  (Appendix  H) . 

TABLE  VIII 

[ ANOVA: Consistency  Measure  by  Cl  and  Zeta 


Source  of  Variation 

Sum  Sqrs 

df 

Mean  Sqr 

F 

Si gni f 

Measure (ci , zeta) 

.090 

1 

.090 

3.  14 

.078 

Residual 

7.  184 

250 

.029 

Total 

7.274 

251 

.029 

This  test  shows  that  when  Cl  and  Zeta  are  combined,  that 
the  type  (Cl  or  zeta)  of  consistency  measure  contributes  a 
significant  amount  of  the  variation  in  the  combined  measures. 


_ _ TABLE  VII _ 

Correlation  Between  Bottom  Level  Priority  Vectors 
Kendal  l^'s  Tau  (Spearman’s  Rho  in  parens) 


EigVM 

VoteM 

Battel 

EqWts 

EigenGM 

.847 

.821 

.549 

.399 

( -  962) 

(-941) 

(.716) 

( . 530) 

Ei genVM 

- 

.858 

.587 

.394 

— 

(.960) 

(.743) 

(.522) 

VoteV 

- 

- 

.545 

.404 

- 

- 

(.689) 

( . 530) 

Battel 

— 

- 

- 

.629 

— 

— 

- 

(.750) 

Where: 

EigGM:  Weights  -from  geometric  mean  input  comparison 

EigVM:  Weights  -from  average  ot  judges’  Eigenvectors 

VoteM:  Weights  ^rom  “Voting"  comparison  Matrix 

Battel:  Original  Battel le  Weights  (Revised  Hierarchy) 
Sig:  All  significances  of  TAU  and  rho  <  .005 


TABLE  VI 


Bottom 

Pri  ori  ty 

Vector 

t  100 

EigenGM 

,  Ei genVM 

,  VoteM. 

Battel , 

and  Equal 

Weight' 

□bj 

EigenGM 

Ei genVM 

VoteM 

Battel 

Equal  Wt! 

1 

6.3940 

6.4167 

4.7310 

3.54 

2.083 

2 

3.7252 

3.4867 

3.4113 

3.54 

2.083 

3 

1 . 6958 

1.9631 

1.7679 

3.54 

2.083 

4 

4.0310 

4. 4536 

3.9591 

3.54 

2.083 

5 

4.  1700 

4.3657 

3. 8595 

3.35 

2.083 

6 

0. 8896 

0. 9962 

O. 7968 

3.35 

2.083 

7 

2.5854 

3. 4281 

2.3904 

3.35 

2.083 

a 

4.2812 

4. 1606 

3.9591 

3.54 

2.083 

9 

12.8607 

10.6714 

12.4500 

4.877 

5.556 

10 

4.7603 

4.0542 

3.8346 

4.877 

5.556 

11 

8. 6790 

8.  5744 

8.6154 

4.877 

5.556 

12 

1 2 . 3000 

12.8000 

13.6000 

14.823 

16.667 

13 

3.0702 

2. 6220 

1 . 9885 

3.  19 

5.556 

14 

4.0902 

3.3304 

3.9770 

6.  39 

5.556 

15 

3.0396 

3.2568 

3. 7345 

3.  19 

5.556 

16 

0.6572 

1 . 2440 

1.2977 

2.26 

1.852 

17 

0.6521 

0. 6950 

0.6714 

1.13 

1.852 

18 

1 . 2086 

1 . 2376 

1.7778 

1.13 

1.852 

19 

1 . 7440 

1.9311 

1 . 8753 

5.  52 

5.556 

20 

0.5510 

0.9051 

0.5449 

1.13 

1.389 

21 

2-0711 

1.9323 

2. 3099 

1.  13 

1 . 389 

22 

2. 4495 

2. 6075 

2.8010 

1.13 

1.389 

23 

1 . 5666 

1 . 7455 

2-0183 

1.  13 

1.389 

24 

0. 7500 

0. 9694 

0. 9928 

1.96 

2.  778 

25 

1.5750 

2. 2008 

2.0944 

1.96 

2.  778 

26 

1 . 7375 

2.2139 

2. 1624 

1.96 

2.778 

27 

1 . 3500 

1.5196 

1.9448 

1.96 

2.  778 

28 

1 . 1 500 

1.0611 

1.3192 

1.96 

2.778 

29 

1.9875 

1 . 8209 

2.0672 

1.96 

2.  778 

30 

1.7183 

1 . 5206 

1.3164 

0.  653 

0.926 

31 

1.6195 

1. 1646 

1.1986 

0.  653 

0.926 

32 

0.6123 

0.6422 

0. 5042 

0.653 

0.926 

Where: 

EigGM:  Eigen  vector  calculated  from  the  geometric  mean  input 
EigVM:  Eigen  vector  from  the  arithmetic  average  judges’  vectors 
VoteM:  Priority  vector  from  "voting"  pairwise  comparison  matrix 

Battel:  Original  Battel le  weights  applied  to  revised  hierarchy 
Equal:  Each  subobjective  contributes  an  equal  portion  to 

the  next  higher  objective  in  hierarchy. 
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FIGURE  5 


Signi-f icance  of  Tau  Between  Priority  Vectors 
as  a  Function  of  N 
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Rank  is  the  eigenvector  derived  from  the 
geometric  mean  of  the  12  judges"  inputs. 

Rank  is  the  vector  mean  (arithmetic)  of 
the  12  judges"  eigenvectors. 

Rank  is  the  vector  mean  (arithmetic)  of  the 
12  judges"  voting  vectors,  i.e.,  it  is  the 
proportion  of  votes  received  for  each  objective. 
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FIGURE  4 
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Rank  is  derived  from  the  eigenvector  from 
the  geometric  mean  of  the  12  judges’  inputs. 
Rank  is  derived  from  the  mean  (arithmetic) 
of  the  12  judges  eigenvectors. 

Rank  is  the  arithmetic  mean  of  the  12  judges’ 
voting  vectors,  i . e. ,  the  mean  vector  is 
comprised  of  the  proportions  of  votes 
received  by  each  objective. 


The  general  trend  for  the  AHP  vectors  established  in  the 
first  level  holds  at  the  bottom  level.  The  geometric  mean 
input  is  closer  to  the  vector  mean  than  the  vote  mean.  The 
importance  of  this  correlation  is  in  the  closeness  of  the 
vote  mean  to  the  geometric  mean  and  vector  mean.  It  shows 
that  a  ratio  scaled  input  converted  to  an  ordinal  input  can 
reflect  the  strength  of  preference.  This  will  be  further 
discussed  in  Chapter  V*l,  Conclusions. 

Besides  the  mathmatical  aggregation  techniques,  this 
thesis  sought  to  apply  interpersonal  value  comparisons  in  the 
form  of  individual  self  evaluations.  However,  there  was 
neglible  discrimination  among  the  jduges  so  this  examination 
was  discontinued.  All  but  one  of  the  108  self  evaluations 
rated  themselves  "good"  or  better.  The  single  ''poor”  self 
evaluation  was  judge  #3  on  the  logistics  criterion.  Because 
his  comparisons  were  close  to  the  mean,  there  would  be  no 
additional  insight  gained  by  deemphasizing  his  inputs. 


Aqqregati on 

The  aggregation  analysis  sho.  j'1  that  the  three  different 
aggregation  methods:  geometric  mean  input,  eigen  vector 
arithmetic  mean,  and  the  vote  mean,  all  resulted  in 
statistically  identical  priorities.  Both  the  top  level  and 
the  bottom  level  priority  vectors  were  examined. 

The  similarity  between  the  three  aggregation  methods  for 
the  top  level  is  graphically  seen  in  Figure  4,  which  displays 
a  histogram  of  the  top  level  weights.  The  significance  of 
Kendall’s  tau  between  these  priority  vectors  is  displayed  as 
a  function  of  the  size  of  the  matrix  in  Figure  5.  Note  that 
perfect  aggreement  for  three  ranks  is  only  significant  to 
.118  because  of  the  small  dimension  of  the  matrix. 

The  similarity  for  the  bottom  level  priorities  was 
extended  to  include  the  priorities  from  equal  weights 
throughout  the  hierarchy  and  the  original  Battelle  ranks  (for 
the  reduced  number  of  criteria  in  the  revised  hierarchy). 
Table  X  shows  the  bottom  vectors  and  Table  XI  shows  the 
correlation  (tau)  between  the  different  vectors. 

Table  XI  shows  that  all  of  the  vectors  statistically 
correlated  (siq.=.005).  It  can  be  seen  that  AHP  and  the 
voting  vectors  have  less  correlation  with  equal  weights 
vector  than  Battelle.  This  is  expected  as  the  Battelle 
hierarchy  was  closer  to  equal  weights  because  it  elicited 
fewer  weights. 


had  a  small  range.  The  small  changes  in  the  ratings  caused 
the  rank  order  to  shift  in  the  Delphi  iteration  (Figure  3). 
This  supports  Eckenrode’s  findings  that  direct  assessment 
ratings  result  in  priority  vectors  that  have  small  ranges  and 
large  variances.  It  also  helps  explain  Nhy  there  is  such  a 
wide  correlation  difference  between  the  AHP  priority  vector 
and  two  Delphi  iterations. 

_ _ TABLE  V _ 

Correlation  (tau)  Between  Top  Level  Priorities 
AHP  Vectors  and  Direct  Assessment 

EiqVh  Voteh  Del  phi O 

EigGM  1.0  .931  .733 

EigVM  -  .931  .733 

VoteM  -  -  . 788 

Del  phi  0  -  -  - 

Where: 

EigGM:  Rank  is  the  eigenvector  derived  from  the 
geometric  mean  of  the  12  judges”  inputs. 

EigVM:  Rank  is  the  vector  mean  (arithmetic)  of 
the  12  judges”  eigenvectors. 

VoteM:  Rank  is  the  vector  mean  (arithmetic)  of  the 
12  judges”  voting  vectors,  i.e.,  it  is  the 
proportion  of  votes  received  for  each  objective. 
OelphiO:  Rank  of  the  original  Battelle  Weights 
Delphi  1:  Rank  of  the  first  Delphi  iteration  of  the 
Battelle  weights 


Del  phi  1 
.600 
.600 
.645 
.867 


correlated,  what  is  important  is  the  magnitude  o-f  the  change 
in  correlation.  Note  that  the  -final  column  shows  the 
hypothetical  situation  where  there  were  equal  weights  given 
to  all  the  criterion.  The  lowest  correlation  is  with  the 
geometric  mean  o-f  the  inputs.  The  highest  correlation  is  with 
the  Battel le  scores,  which  is  expected  because  Battel le,  had 
the  fewest  weights.  They  weighted  the  top  level  plus  three  of 
the  32  revised  hierarchy  criterion. 

The  tau’s  in  the  *'NoWts“  column  are  inversed  from  the 
equal  weights  column.  With  no  weights,  the  importance  of  a 
given  objective  is  directly  proportional  to  the  number  of 
subobjectives  that  constitute  it  (the  larger  the  sum).  It  is 
only  by  coincidence  that  the  number  of  these  subobjectives  is 
more  closely  aligned  to  the  weights  gained  throughout  the 
heirarchy  (AHP)  than  to  weights  applied  to  just  the  top  level 
(Battel le).  The  "NoWts"  column  is  included  for  completeness 
and  its  application  is  limited  to  the  data  in  this  survey. 


TABLE  XII 


TAV  Design  Scores 
By  Weighting  Schemes 


TAV 

Ei  gGM 

EigVM 

VoteM 

Battel 

Equal 

NoWts 

1 

488 

492 

499 

504 

558 

583 

2 

520 

525 

532 

553 

598 

541 

t; 

537 

539 

551 

577 

632 

576 

4 

705 

696 

699 

633 

649 

697 

5 

686 

687 

688 

662 

669 

586 

6 

587 

598 

599 

620 

653 

574 

7 

703 

707 

708 

692 

716 

636 

8 

741 

745 

740 

729 

741 

664 

9 

616 

629 

623 

606 

643 

664 

10 

544 

558 

548 

555 

592 

556 

11 

569 

577 

568 

555 

588 

572 

12 

621 

631 

619 

650 

651 

619 

13 

597 

610 

597 

618 

619 

623 

14 

686 

694 

682 

690 

690 

666 

Where: 

EiqSh:  Eigen  vector  calculated  from  the  geometric  mean  input 

EigVM:  Eigen  vector  from  the  arithmetic  average  judges’  vectors 

VoteM:  Priority  vector  from  "voting"  pairwise  comparison  matrix 

Battel:  Original  Battelle  weights  applied  to  revised  hierarchy 
Eaual :  Each  subabjective  contributes  an  equal  portion  to 

the  next  higher  objective  in  hierarchy. 

NoUts:  No  weights  applied  to  Revised  Hierarchy 
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TABLE  XIII 

Correlation  Between  Final  Scores 
Kendall’s  Tau  (Spearman’s  Rho  in  parens) 


EigVM 

VoteM 

Battel 

NoWts 

EqWts 

Ei genSM 

.972 

.906 

.790 

.611 

.685 

(.995) 

(.981) 

(.923) 

(.804) 

(.841) 

EigeVM 

- 

.912 

.818 

.597 

.714 

- 

(.982) 

(.942) 

(.799) 

(.864) 

VoteM 

- 

- 

.818 

.597 

.714 

— 

- 

(.931) 

(.799) 

(.886) 

Battel 

- 

- 

- 

.500-.014- 

.862 

— 

- 

- 

(.691) 

(.959) 

NoUts 

- 

- 

- 

- 

.398- 

Where: 

(.607) 

EigGh:  Weights  from  geometric  mean  input  comparison 

EigVM:  Weights  from  average  of  judges’  Eigen  Vectors 

Votetl:  Weights  from  "Voting'*  comparison  Matrix 

Battel:  Original  Battel le  Weights  (Revised  Hierarchy) 

NoWts:  No  weights  applied:  score  was  sum  of  attribute  values 
-Sig-s  If  the  significance  of  TAU  (=  Significance  of  rho) 
is  greater  .01  then  sig  shown  next  to  tau  -xxx- 


where: 


Battel le  Original  Weights  Applied  to  Revised  Hierarchy 
Q  AHP  (Geometric  Mean) 
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7 


4  14 


9  12  13 

TAV  Number 


AHP  (Geometric  Mean) 
Equal  Weights 
No  Weights 


Chapter  Vt.  Conclusions 


Introduction, 

The  conclusion  discussion  Mill  'fall  into  three 
categories.  The  first  category  are  the  formal  conclusions 
drawn  from  the  research  purpose,  to  analyze  the  three  facets 
of  the  AHPa  feedback,  aggregation  methods,  and  consistency 
measures.  The  second  category  of  conclusions  consist  of 
lessons  learned  applying  AHP  by  survey.  Finally,  the  last 
division  of  conclusions  are  suggestions  for  follow-on 
research. 

Feedback 

The  feedback  analysis  was  limited  to  the  direct 
assessment  portion  of  the  study  (Battel le>  because  it  could 
not  be  determined  that  any  of  the  changes  in  the  AHP  portion 
of  the  survey  were  due  to  subjects  reading  the  "Quality 
Controlled  Feedback."  However,  the  top  level  priority  vectors 
derived  from  direct  assessment  showed  both  a  variance 
reduction  and  increase  in  concordance 

after  the  judges  had  received  the  means  from  the  first 
survey.  This  reflected  that  there  was  more  agreement  after 
the  feedback  than  before  the  feedback. 

There  was  one  unexpected  outcome.  The  paired  comparison 
results,  both  the  voting  vector  and  the  eigenvector  for  the 
top  level  objectives  correlated  closer  to  the  original 


Battel le  survey  than  to  the  second  Del phi -Battel le  survey. 
This  is  renarkable  considering  the  Del phi -Battel le  survey  was 
accomplished  simultaneously  with  the  first  AHP  survey,  while 
the  original  Battel le  survey  was  accomplished  over  one  year 
earlier.  Perhaps  the  explanation  is  that  the  feedback  did 
force  an  unnatural,  "bandwagon,"  variance  reduction  while  the 
AHP  elicited  the  "true"  opinions  that  came  closer  to  matching 
the  original  survey. 

A  word  of  caution  is  in  order.  Both  the  original  and 
second  Del  phi -Battel le  priority  vector  values  had  very  narrow 
ranges  and  large  variances  between  judges.  These  are  the 
characteristics  that  Eckenrode  predicted  for  vectors  derived 
from  direct  assessment,  which  makes  them  poor  techniques  for 
discrimination  purposes.  These  same  characteristics  allow 
very  minor  shifts  to  alter  the  rank  order  which  reduces  the 
tau  correlation  between  vectors.  The  graphical  display  in 
Figure  3  shows  much  less  of  the  correlation  drop  between  the 
iterations  as  is  shown  statistically  in  Table  111. 
Aggregatign_Methgds 

All  three  of  the  aggregation  methods  resulted  in 
graphically  and  statistically  the  same  priority  vectors. 
Ellsburg  mentioned  that,  in  general,  converting  interval 
comparisons  to  an  ordinal  scale  does  not  preserve  the 
interval  (Ellsburg,  1954s 228).  Graphically  and  with  rank 
ordered  statistics,  this  research  showed  that  the  interval 
was  well  preserved  when  converting  AHP  matrices  into  voting 


matrices.  However,  there  may  be  untested  boundaries  in  the 
number  o4  judges  or  size  q4  the  matrix  that  exist  when  this 
relationship  no  longer  exists. 

There  is  another  word  of  caution  about  the  close 
relationship  between  the  ordinal  (voting)  and  ratio  (AHP) 
priority  vectors.  The  "voting"  matrix  was  not  elicited 
separately  from  the  AHP  inputs.  The  AHP  ratio  inputs  were 
converted  to  votes  (preference  direction  only).  This 
conversion  assumes  that  the  judge  would  state  the  same  strict 
preferences  as  he  had  with  the  ratio  scale. 

This  assumption  may  not  be  valid.  Further  research  may 
show  that  when  going  from  a  continuous  rttio  scale  (-9  to  +9) 
to  a  three  level  ordinal  scale  (O, 1/2, 1),  the  indifferent 
ordinal  point  may  absorb  many  of  the  comparisons  previously 
judged  "weakly"  more  important.  That  is,  the  judge  would 
rather  rate  a  weak  preference  as  indifferent  when  his  only 
options  are  strict  preference  or  indifference. 

The  voting  matrix  should  be  elicited  separately  from  the 
AHP  matrix  to  obtain  a  valid  indication  of  the  correlation 
between  these  two  methods.  This  thesis  showed  that  when  the 
voting  matrix  is  generated  from  the  AHP  matrix,  there  is  a 
strong  correlation  between  the  resultant  priority  vectors  and 
that  the  vectors  were  graphically  very  similar. 

QQQSistency 

The  ratio  consistency  measure.  Cl,  is  highly  correlated 
with  the  ordinal  consistency  measure,  zeta.  Both  measures 


117 


re-flect  the  inconsistency  in  a  matrix  of  pairwise 
comparisons.  The  ordinal  measure,  zeta  is  not  a-ffected  by  the 
matrix  dimension  (number  o-f  objectives  being  compared),  as 
much  as  ratio  consistency  is.  One  reason  may  be  that  the 
matrix  dimension  determines  the  exponential  -factor  for  the 
iterative  solution  to  the  eigenvalue  solution  while  the  zeta 
solution  is  a  ratio  of  two  third  order  equations.  That  is,  a 
plot  of  the  consistency  index  versus  the  matrix  dimension 
results  in  an  exponential  decay.  When  the  dimension  >  10 
there  is  almost  perfect  consistency,  less  than  .05  (Saaty, 
1980:63) . 

Zeta,  although  its  significance  is  solely  dependent  on 
the  dimension  as  seen  in  Appendix  H,  is  not  as  influenced  by 
the  matrix  size.  Zeta  does  not  indicate  ratio  consistency, 
but  the  close  correlation  (significance  =  .001)  to  the  Cl 
shows  that  it  is  an  equivalent  measure. 

BattBjLle_Resul.ts 

The  fact  that  there  were  only  minor  shifts  in  the  TAV 
design  rankings  when  the  Battel le  weights  were  applied  to  the 
original  and  revised  hierarchies  shows  that  the  discarded 
elements  had  little  impact  on  most  of  the  TAV  designs.  The 
actual  differences  between  applying  Battel  le-'s  weights  for 
the  entire  hierarchy  or  just  for  the  revised  hierarchy  is 
actually  irrelevant  for  this  thesis,  because  this  scope  is 
limited  to  comparing  techniques  on  a  fixed  hierarchy  size. 

One  could  have  accomplished  the  study  with  any  fixed  size 
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hierarchy. 

Applying  the  weights  through  out  the  hierarchy  (AHP) 
shifted  the  -following  priorities  -from  the  Battel  le 
priorities.  Consequently,  the  “value"  to  the  designs 
strong/weak  in  these  objectives  shifted: 

Emphasis  Increased: 

#9:  Payload  Mass 

#11:  Trajectory  Flexibility 

#12:  Rapid  Response 

De-emphasis: 

#24:  Loiter 

#20:  #  Jets  (Maintainability) 

#19:  #  Rockets  (Maintainability) 

#6:  Single  Stage 

#3:  Horizontal  Take  Off /Land 

Suryey_AQeli cation 

Much  insight  was  gained  on  what  should  be  included  in  a 
survey  application  of  the  AHP.  While  Saaty  promotes  group 
interactions  with  a  knowledgeable  monitor,  this  is  often  not 
practical.  The  alternative  is  to  mail  the  AHP  solicitation  to 
the  "experts."  As  such,  the  insight  gained  in  this 
application  may  be  beneficial  to  others. 

The  matrix  format  used  in  this  study  to  obtain  the 
pairwise  comparisons  was  selected  over  the  continuous  line 
suggestion  by  Saaty  (1980:35).  The  plan  was  to  reduce  the 
"business”  and  the  bulk  of  the  survey.  This  approach  should 
be  avoided.  The  small  savings  in  display  bulk  was  more  than 
compensated  for  by  the  additional  instruction  to  insure  the 
comparisons  were  not  inverted  when  making  the  judgement. 
Although  less  than  V/.  of  the  returned  comparisons  had  been 
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inverted  (sign  errors),  there  would  probably  be  far  fewer 
errors  on  the  continuous  line  format. 

Scale..  The  scale  used  in  the  solicitation  should  be 
displayed  differently  from  the  Saaty  presentation  (1980:54). 
The  scale  proved  sound  and  very  workable  but  this  thesis 
found  that  there  was  an  indication  that  some  of  the 
respondents  treated  the  scale  as  interval  rather  than  ratio. 
For  example,  suppose  that  a  judge  felt  objective  A  was  three 
units  more  important  than  objective  B  and  objective  B  was  two 
units  more  important  than  C  which  had  a  value  of  one  unit.  If 
the  judge  put  these  interval  responses  into  the  AHP  matrix, 
the  AHP  "ratio"  priority  vector  would  be  different  from  his 
"true"  "interval"  priority  vector: 

Priority  Vectors  based  on  Judge’s  Scale 

Interval  vs  Ratio  (AHP) 

A=6  =>  1.00  =>  1.00 

B=3  =>  .50  =>  .35 

C=1  =>  .17  =>  .19 

This  study  also  found  that  the  respondents  were 
reluctant  to  use  other  than  odd  integers  when  given  the  Saaty 
format  (Saaty,  1980:54).  Inaddition,  no  one  responded  with  a 
fraction  of  a  number,  although  the  scale  is  continuous 
between  one  and  nine.  Thus,  they  were  unable  to  achieve  a 
full  range  of  weights. 

The  discontinuous  jumps  in  the  priority  vector  were 
especially  evident  when  there  were  less  than  five  objectives 
being  compared  because  the  few  comparisons  resulted  in 
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incremental  jumps  in  the  resulting  priority  vector  elements. 
This  dissatisfied  at  least  tMO  judges  Mho  sought  weights 
between  .5  and  1,  but  did  not  use  a  ratio  rating  between  one 
and  two.  Only  after  one  respondent  was  informed  that  ''weak" 
importance  meant  three  times  the  importance  of  the  other 
criteria,  did  he  realize  he  overrated  the  objective  and  so  he 
evaluated  the  comparison  as  1.25  as  important. 

Saaty  has  described  the  derivation  of  the  1-9  ratio 
scale  (Saaty,  1980:53-66).  The  scale  is  excellent  for  large 
matrices  where  the  numerous  comparisons  wash  out  the 
incremental  jumps  (at  some  cost  to  consistency).  However, 
this  research  found  that  his  interval  portrayal  of  the  scale 
distorted  the  weights,  especially  for  the  smaller  matrices. 

The  author  believes  few  people  judge  less  than  "weakly" 
more  important  as  twice  as  important,  or  interpret  "weak  to 
absolute"  comparisons  on  a  ratio  scale.  In  order  to  better 
represent  the  subjective  values  of  the  expert,  the 
questionnaire  should  reflect  the  ratio  nature  of  the  scale: 

Saaty  Presentation: 

Abso  VeryStrg  Strong  Weak  Equal  Weak  /  VeryStrg  Abso 
Obj  1;_1 _ i _ I _ I _ I _ ^  * _ L_  Qbj  2 

Revised  Presentation: 

Abso  Abso 

VeryStrong  VeryStrong 

Strong  Strong 

Weak  Equal  Weak 

Obj  1  !  !  ! _ I _ 1 _ i _ !  !  ! 

The  revised  presentation  has  two  major  advantages  for 
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eliciting  weights  by  survey.  First,  it  more  accurately 
represents  the  e-ffect  o-f  the  ratings  —  the  logarithmic  versus 
additive  e-ffect  that  Saaty’s  presentation  implies.  Secondly, 
it  graphically  shows  a  continuous  scale  which  is  asymptotic 
at  either  end.  The  asymptote  (“absolute")  emphasizes  that  the 
comparisons  roust  be  within  the  same  order  of  magnitude  to 
give  valid  weights. 

EyCther_Research 

The  discrimination,  or  distance  between  objective 
weights  in  the  resulting  priority  vector  is  another  area 
requiring  further  research.  The  statistical  evaluations  used 
in  this  research  tested  rank  order  concordance  (M)  and 
correlation  (tau) .  These  tests  fail  to  account  for  the 
distance  between  the  ranks,  which  may  be  substantially 
different  between  aggregation  methods. 

This  research  appealed  to  graphical  displays  to  show  the 
distance  similarity  (Figures  III  and  IV).  Graphs  cannot  prove 
relationships  or  provide  the  sensitivity  analvsis  necessary 
to  establish  the  boundaries  where  the  strength  of  preference 
IS  preserved.  For  example,  the  size  of  the  matrix  mav  have  to 
be  greater  than  six  to  to  preserve  the  strength  of  preference 
when  converting  the  AHP  to  a  "voting''  matrix. 

Further  research  is  necessary  to  defter  mine  the  effect 
that  the  revised  presentation  would  have  on  the  consistency 
and  range  of  the  priority  yectors.  The  author  suspects  that 
consistency  would  improve  while  the  range,  and  thus  the 


discriminating  power,  ai  the  priority  vector  would  diminish. 

Consistency  should  improve,  especially  in  smaller 
matrices  (n  <  5)  because  “true"  ratings  between  one  and  two 
would  not  require  that  the  matrix  be  inconsistent,  as  what 
happens  with  integer  ratings.  For  example,  i-f  the  respondent 
limited  himsel-f  to  integer  values  and  his  "true"  normalized 
priorities  for  options  A,  B,  and  C  were  1,.6,  and  .3  his  best 
choices  for  the  AHP  matrix  would  be: 


"True" 

A 

B 

C 

"AHP 

1.0 

A  - 

2 

3 

1.00 

.6 

B  - 

- 

2 

.55 

.3 

C  - 

.30 

The  comparsions 

above 

woul  d 

resul t 

Morse,  there  is  a  divergence  of  (.6-. 55)  =  .05  from  the 
"true"  normalized  vector.  However,  if  the  judge  made 
continuous  ratings  from  the  revised  scale  he  would  have: 

A  B  C  "AHP" 

A  -  1.7  3.3  1.00 

B  -  -  2.0  .59 

C  -  -  -  .31 

The  matrix  above  gives  a  Cl  =  .0001  and  a  divergence 
from  the  "true"  vector  =  .01.  The  judge  may  still  give 
integer  ratings  with  the  revised  presentation,  but  the  author 
feels  the  logarithmic  intervals  makes  this  less  likely. 

Another  advantage  of  the  logarithmic  presentation  is 
that  the  judge  may  be  more  aware  the  importance  of  all  the 
attributes  being  on  the  same  order  of  magnitude.  The  clear 


APPENDIX  Ai  Initial  Th«*i«  Survay 


The  following  comprises  the  criteria  for  survivability; 

t niter-  Space  -  Rated  lower  if  need  to  reduce  payload  (OMS  Propellent) 

'  Atmosphere  -  Rated  lower  if  single  stage 

Mobile  Launch:  Rated  lower  if  required  long  runways  and/or  cryogenics 

Fsraoe-  Rated  lower  if  required  large  carrier  aircraft 
Base  Escape.  R^te^^^^  once  launch  ordered) 

Dispersal  Range:  Rated  1.0  if  greater  than  1000  miles  else  given  .8-.9  for  300-600  miles 

Laser  Hardness:  Rated  1.0  If  tile  TPS  and  .5  if  metal  TPS 

Maneuverability:  Below 

1  -  Equal  Importance 
3  -  Weak  Importance 

5  -  Moderate  Importance  (Column  row  ■♦•) 

7  -  Strong  Importance 
9  -  Absolute  Importance 


Which  factor  ccntrlbuCes  more  to  TAV  maneuverability? 
(column  row  ■►) 


SPACE  (OMS 
AERODYNAMIC 


*  ^ 
i/ 


MANEUVERABILITY  /| 

4’  e 


♦  propellents) 

(L/Drag  ♦TPS)  _ 


Which  factor  contributes  more  to  TAV  survivability? 
(column  row  -f) 


SURVIVABILITY 


I 

^  ,V  -S 

^  fa  S!  Q 
^  ^  ^  ^  k  S 


LOITER 

MOBILE  UUNCH 
BASE  ESCAPE 
DISPERSAL  RANGE 
LASER  HARDNESS 


APPENDIX  A:  Initial  Thasis  Survay 


The  following  comprise  the  criteria  for  logistics  (supportability ) . 


1  -  Equal  Importance 
3  -  Weak  Importance 

5  -  Moderate  Importance  (Column  row  +) 
7  -  Strong  Importance 
9  ~  Absolute  Importance 


Facilities:  Rated  lower  if  large  hangers  and/or  complex  mate/demate  apparatus 


Circle  your  level  of  knowledge/experience  with  logistics 
Minimal  Average  Good 
123  456789  10 
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^^^^ENDIX  Ai  Initial  Ttiasis  Survey 


The  following  comprise  the  factors  for  measuring  TAV  technological  -isk. 


1  -  Equal  Importance 
3  -  Weak  Importance 
5  -  Moderate  Importance 
7  -  Strong  Importance 
9  -  Absolute  Importance 


(Column  row  +) 


Structure:  Higher  risk  if  hot  structure,  metal  matrix,  advanced  ceramic  TPS 

Mass  Fraction:  Higher  risk  if  greater  fraction  than  Shuttle  (.87) 

Propulsion:  ABES  -  generally  lower  risk,  except  turbo  ramjet 

ROCKETS  -  disparity  from  shuttle  engines  increased  risk 


Which  contributes  more  to 
technological  risk?  (column  -,  row  +) 


RISKS  - 


PROPULSION 

STRUCTURE 


Circle  your  level  of  knowledge/experience  with  these  factors  of  Technological  risk 
Minimal  Average  Good 
123456789  10 
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APPENDIX  A:  Initial  Thasis  Survey 


The  following  comprise  the  criteria  for  propellants  and  maintainst-iity . 


1  -  Equal  Importance 
3  -  Weak  Importance 
5  -  Moderate  Importance 
7  -  Strong  Importance 
9  -  Absolute  Importance 


(Column 


row  +) 


Which  criterion  is  more  important  with 
regard  to  propellant  logistics  (supportability )?  ^ 


A  ^ 


PROPELLANTS 

AMOUNT (to  deliver  equivalent  mass) 
NUMBER  OF  DIF  TYPES 


#  (5^ 


Circle  your  level  of  knowledge/experience  wlt)5  propellants 
Minimal  Average  Good 
123  A567091O 


Which  faccof,  as  ic  decreases  in  nuotber  or  complexity, 
contributes  more  to  TAV  maintainability? 

(column  row  +) 


maintainability 

number  of  JETS 
number  of  rockets 
TPS  rypE 


^  § 


Circle  your  level  of  knowledge/experience  with  maintenance 
Minimal  Average  Ck>od 

123  A567  89  10 
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APPENDIX  As  Initial  Th««i«  Survay 


The  following  comprise  the  factors  for  measuring  TAV  perfomsr.cc . 


1  -  Equal  Importance 
3  -  Weak  Importance 

5  -  Moderate  Importance  (Column  row  +) 
7  -  Strong  Importance 
9  -  Absolute  Importance 


Trajectory:  Ability  to  alter  mission  after  launch:  Based  on  a  baseline  value  plus  a  value 
if  powered  reentry  and  minus  a  value  if  expendable  equipment. 

Payload:  Mass  deliverable  on  a  90°  low  (87m)  orbit 


Which  factor  is  a  better  measure  of  TAV 

> 

performance?  (column  row  +) 

PERFORMANCE  / 

1  ^  V 

PAYLOAD  1 

VOLUME 

y 

_! 

Circle  your  level  of  knowledge/experience  with  your  knowledge  of  performance. 
Minimal  Average  Good 

123  A567  89  10 


appendix  A:  Initial  Thesis  Survey 


The  comparisons  may  seem  rather  obtuse  at  times:  "  Which  is  more 
important,  an  apple  or  an  orange  with  regard  to  taste?"  -  but  this 
series  of  comparisons  can  reliably  determine  the  overall  priorities, 
or  weights,  of  the  decision  maker.  However,  it  is  important  for  you, 
as  the  "expert  decision  maker,"  to  expeditiously  accomplish  these 
comparisons  without  looking  back  to  correct  for  consistency.  The  final 
results  will  be  returned  to  you  to  verify  that  the  weights  reflect  what 
you  intended. 

Inorder  to  have  a  common  denominator  for  the  comparisons,  the 
scenerio  is  arbitrarily  set  as  a  general  world  conflict.  The  TAV's 
mission  is  multi-purpose:  weapon  delivery  (space  and  ground)  and  recoti- 
naissance.  Furthermore,  the  TAV  is  threatened  by  the  same  threats  that 
face  our  21st  century  triad  forces:  preemptive  strikes,  terrorists, 
ground  and  space-based  lasers,  nuclear  -  conventional  weapons,  particle 
beams,  etc. 

For  the  more  technical  criteria,  a  self-assessment  question  is 
presented.  Your  answer  will  provide  feedback  about  how  confident  you 
feel  in  that  particular  area.  Again,  thank  you  for  your  cooperation. 

Any  comments  you  may  have  will  be  appreciated. 
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APPENDIX  As  Initial  Thasis  Survay 


The  opposite  table  shows  the  values  you  should  assign  the  compar¬ 
isons  between  elements.  For  example,  with  regard  to  TAV  maneuverability, 
if  you  feel  that  space  maneuverability  is  weakly  more  important  than 
aerodynamic  maneuverability,  then  you  would  give  the  first  cell  a  '3'. 
(Note:  space  maneuverability  is  measured  by  whether  the  system  has  QMS 
and  the  necessaary  propellents,  while  aero  maneuverability  is  measured 
by  the  lift  to  drag  ratio  and  the  ability  to  withstand  re-entry  tempera¬ 
tures).  If  you  feel  that  the  column,  "powered  re-entry,"  is  "strongly" 
more  important  than  either  of  the  row  elements:  Space  or  Aero  Maneuvering, 
you  would  put  "-7"  in  the  two  remaining  cells.  Remember,  the  cell  is 
positive  if  the  row  titles  are  more  important  and  negative  if  the  column 
titles  are  more  Important.  The  following  figure  illustrates  the  completed 
matrix: 


Which  factor  contributes  more  to  TAV  maneuverability? 
(column  -,  row  +) 


s  J' 


-  # 


•f 


<8^ 


MANEUVERABILITY 

SPACE  (QMS  ♦  propellents) 
AERODYNAMIC  (L/Drag  ♦  TPS) 

'^Vavos  coVtnr'K  “WZ 
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APPENDIX  Ai  Initial  Thasis  Survay 


COMPARISON  SCALE 


INTENSITY 

OF  IMPORTANCE 

DEFINITION 

EXPLANATION 

1 

Equal  Importance  in  both 
elements 

Two  elements  contribute  equally 
to  the  property 

3 

Weak  Importance  of  row  element 
over  column  element 

Experience  and  Judgment  slightly 
favor  row  over  column  element 

S 

Moderate  importance  of  row 
over  column  element 

Experience  and  Judgment  moderately 
favor  row  over  column  element 

7 

Strong  or  essential  importance 
of  row  over  column  element 

Experience  and  Judgment  strongly 
favor  row  over  column  element 

9 

Absolute  or  demonstrated  impor¬ 
tance  of  row  over  column  element 

The  evidence  favoring  the  row 
element  is  of  the  highest  possible 
order  of  affirmation 

2, 4, 6, 8 

Intermediate  values  between 
two  Judgements 

Compromise  is  needed  between 
two  Judgments 

Negative:  The  inverse  relationship  holds.  The  column  element  is  more  important 

than  the  row  element  with  regard  to  the  given  property. 
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INSTRUCTIONS  PART  II 

Part  II  hierarchy  is  a  subset  of  the  original  Battelle  hierarchy. 

1  combined  Requirements  and  Goals  (Feedback  from  the  original  Battelle 
survey)  and  eliminated  those  criteria  that  failed  to  distinguish 
between  design  proposals.  These  criteria  were;  manned  flight,  few 
orbits,  launch  within  5  minutes,  and  two  missions  per  day.  Multiple 
dispersal  sites,  a  criterion  under  survivability,  was  eliminated  because 
It  was  measured  in  a  similar  way  that  the  criterion,  mobile  launch,'was 
measured  (by  the  runway  length  required).  Finally,  environmental  impact 
was  eliminated  because  of  the  insignificant  weight  given  to  it  in  the 
1983  Battelle  study. 

The  hierarchy  on  the  opposite  page  shows  the  resulting  relationships. 
The  circled  clusters  indicate  the  groups  of  paired  comparisons  that  you 
will  be  asked  to  make  in  the  subsequent  pages. 
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APPENDIX  Ai  Initial  Thasis  Survey 


Part  I 


The  table  below  lists  the  group  average  weights  (1983)  for  the  criteria 
used  to  measure  the  TAV  feasibility/effectiveness.  Also  listed  is  your 
1983  estimate  of  these  weights.  Please  write  in  what  you  now  feel  the 
weights  should  be  based  on  the  same  subjective  1-  10  scale  (least  im¬ 
portant  to  most  important). 


Note:  The  hierarchy  on  the  opposite  page  shows  the  criteria  that  Battelle 
used  to  measure  these  objectives.  If  your  current  estimate  exceeds 
one  standard  deviation  of  the  group  average  (1983),  please  Jot  down 
a  few  words  explaining  your  reasons.  Thank  you. 


REQUIREMENTS  GOALS  PERFORMANCE  COSTS  RISKS  SURVIVABILITY  LOGISTICS  ENVIRON 


average  ; 

9.77 

7.15 

8.08 

’  7.77 

7.00 

7.46 

7.54 

3.69 

standard  dev 

.63 

1.57 

1.38 

1.92 

1.56 

1.66 

1.39 

1.44 

your  1983  input: 

your  1984  input: 

Reasons/coraments : 
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Manngd  Flight 


Global  Ranaa 

Sortia  abarationa 

REQUIREMENTS 

ftapid  Resoonsa 

Rlexibla  Basing 

Horizontal  TU/Landin^ 

^varaa  father  Ops 

Launch  5  flinutg^ 


2  Hisaions/Dav 

GOALS 

Fully  Rausabla 

Sinaia  Staoa 
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payload  WgtQht 
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Trajactory  Flaxlbility 
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RIStfS 

Mass  Fraction 
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DEPARTMENT  OF  THE  AIR  FORCE 

.  AIR  FORCE  institute  OF  TECHNOLOGY  (AU) 
WRIGHT-PATTERSON  AIR  FORCE  BASE.  OH  AMU 


Major  Stalker  Reed 
2506  Cross  Country  RO 
Fairborn,  OH  45324 


irtfRODUCTION 

Thank  you  for  agreeing  over  the  phone  to  assist  bc  1n  iny  thesis 
effort  at  the  School  of  Engineering,  Air  Fifrce  Institute  of  Technology. 

The  purpose  of  ii<y  thesis  is  to  analyze  an  alternative  method  to  evaluate 
multiple  objective  problems.  The  method  Is  called  the  Analytical  Hierarchy 
Process  (AHP),  and  It  establishes  criteria  weights  by  a  series  of  pialr- 
wise  comparisons  between  criteria.  I  am  using  the  1983  Battelle  TAV  study 
as  the  source  of  data.  To  achieve  comparable  findings,  I  need  as  many  of 
the  13  original  TAV  steering  coomlttee  meirbers  as  possible.  In  addition, 
to  reach  valid  conclusions,  I  aeed  your  “expert"  knowledge  and  Insight 
regarding  various  aspects  of  the  TAV.  ,  > 

The  first  page  shows  the  group  weights  for  the  top  level  of  objectives 
from  the  1983  study  and  asks  you  to  update  your  1983  Inputs.  The  subsequent 
pages  apply  AHP  to  the  entire  hierarchy,  which  Is  altered  from  the  original 
Battelle  Hierarchy.  Two  abbreviations  are  used:  OMS  stands  for  Orbital 
Maneuvering  System,  and  TPS  stands  for  the  Thennal  Protection  System 
(Ceramic  tiles  versus  metal). 

The  trial  runs  Indicate  the  survey  should  take  between  25  and  35  minutes. 
I  would  be  very  grateful  If  you  could  return  this  questlonaire,  along  with 
any  insights  that  you  would  like  to  share,  within  a  week  of  receipt.  Again, 
thank  you  for  your  generous  offer  to  assist  ny  thesis  effort. 


Stalker  E.  Reed 
Major,  USAF 
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asymptote  at  “absolutely  more  important"  emphasizes  the 
"extremity"  of  the  comparisons.  Again,  this  does  not  prevent 
objectives  outside  of  one  magnitude  from  being  compared;  the 
scale  only  serves  to  highlight  that  "absolute"  is  absolute. 

The  revised  presentation  may  result  in  less 
discrimination  because  there  may  be  proportionally  more 
ratings  in  the  one  to  two  range  because  the  proportion  of 
space  on  the  scale  may  influence  the  number  of  ratings.  Half 
the  logarithmic  space  is  between  the  ratings,  one  and  two,  sc 
it  is  likely  to  pull  some  judgements  into  this  range  and 
subsequently  reduce  the  discrimination  of  the  resulting 
priority  vector. 

GQQElue[iQfl_E§B§c!<§ 

The  findings  in  this  research  indicate  that  there  is 
very  little  differerence  in  the  three  aggregation  methods 
examined.  Also,  the  AHP  scale  should  be  presented  so  the 
interval  between  ratings  reflects  the  ratio  nature,  i.e.,  it 
shoud  be  logarithmic. 

However,  one  needs  to  exercise  caution  before  applying 
these  results  because  they  are  empirical  results  that  are 
based  on  a  sample  size  of  twelve.  A  larger  sample  size  of 
subjects  who  are  not  high  level  decision  makers,  should 
provide  increased  validity. 
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The  following  matrices  compare  the  criteria  for  measuring  TAV 
requirements/goals  and  the  overall  TAV  feasibility/effectiveness. 


1  -  Equal  Importance 
3  -  Weak  Importance 

5  -  Moderate  Importance  (Column  row  +) 
7  -  Strong  Ir;ortance 
9  -  Absolute  Importance 


Sortie  Operations:  Measures  the  complexity  of  macing  operations 
and  the  numbt-c  of  different  vehicle  elements. 


.f  .h.  TAV, 


REQUIREMENTS/GOALS 


# 

(u  ^  ^  jJ 

^ 


GLOBAL  RANGE 
RAPID  RESPONSE 
FLEXIBLE  BASING 
HORIZONTAL  T.O.  AND  LAND 
ADVERSE  "FATHER 

fully  reusable 

SINGLE  STAGE 
SO.OOOlbs  PAYLOAD 


Costs:  LCC  based  on  50  TAVs  flying  100  flights/year  for  20  years 


Which  element  is  more  important  to  establish  the 
£easiblllty/ef f ectiveness  of  the  TAV  concept? 

(column  tow  +) 

TAV  EFFECTIVENESS  ' 


'1EQUIREMENTS/G0ALS 

PERFORMANCE 

COSTS 

RISKS 

LOGISTICS 
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Major  Stalter  Reed 
Box  4386 
AfTT/HIA 
WPATO,  OH  45433 

TAV  Steering  Oaranittee  Menber,  4  Dec  84 

Thank  you  for  your  speedy  re^nnse  to  ny  thesis  inquiry.  As  premised, 

I  ant  returning  a  few  of  ny  intemiediate  findings.  I  an  very  pleaWd  with 
the  results.  I  have  inclxcsed  a  few  of  the  individual  coiments,  in  case  you 
would  like  to  oonsider  them  in  your  jud^nents.  Note:  the  aaments  are  for 
information  only  and  do  not  reflect  any  results  of  this  stueJV. 

The  prooess  used  in  this  survasy  is  caTIi»d  the  Analytical  Hierarchy 
Ptooess.  It  is  being  used  extensively  in  DOD  to  elicit  %<ei^ting  factors 
from  eiqperts.  A  recent  exanple  is  HQ  tSAF's  *Air  Poroe  2000*  study. . 
Generally,  the  results  are  valid  reflections  of  dedsicn  aukers  opinions. 

I  need  you  to  review  the  weights  this  prooess  derived  from  your 
aao(iarisans  to  insure  they  reflect  your  opirdons.  A  simple  *yes*  or  *no” 
is  all  that  is  neoessary  on  the  respective  pages.  The  original  caiparlscn 
matrioes  are  inclosed  to  allow  you  to  make  any  changes  you  deem  neoessary. 


Please  have  a  blessed  and  warm  Christmas  Holiday 

Stalker  E.  Reed 
Major,  USAF 

PS:  I  would  be  grateful  if  you  oould  return  this  prior  to  the  Christmas  break. 
If  you  would  like  an  executive  sotmary  of  ny  results,  please  circle  belor: 
yes — send  a  suRinary  (Mar-Apr  85} 
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'TVml 


jk/Zr^-WtAjO  ^  o.^ 

Cofe'^b  •  1  ■ 

^vk-tK  cWsJt^  ?(ix  covk^xvkUk^ 


COMPARISON  SCALE 


INTENSITY 

OF  IMPORTANCE  DEFINITION  EXPLANATION 


1 

Equal  importance  in  both 
elements 

Two  elements  contribute  equally 
to  the  property 

3 

Weak  importance  of  row  element 
over  column  element 

Experience  and  judgment  slightly 
favor  row  over  column  element 

5 

Moderate  importance  of  row 
over  column  element 

Experience  and  judgment  moderately 
favor  row  over  column  element 

7 

Strong  or  essential  importance 
of  row  over  column  element 

Experience  and  judgment  strongly 
favor  row  over  column  element 

9 

Absolute  or  demonstrated  impor-  The  evidence  favoring  the  row 
tance  of  row  over  column  element  element  is  of  the  highest  possible 

order  of  affirmation 

!,4,6,8 

Intermediate  values  between 
two  Judgements 

Compromise  is  needed  between 
two  Judgments 

Negative; 

The  inverse  relationship  holds.  The 
than  the  row  element  with  regeu^d  to 

column  element  is  more  important 
the  given  property. 

Arx^Cb"  ^  ^  isoIa^J 

VTNO^AAi.  •  -  '  VlPA'^e  cVt.  '>^ot 

ervwi  >  J  .  sWvVi  e^V  ^ 

lA^ 'M^ 'r(X\  (’t)  C/^t'^Wvy^Ko 


\;^00  QSt 

)t  VTNOvASl 


tfVAi  TM)"^ 

be  cCYicerr^fcl 
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Belcw  are  the  priorities  derived  fran  the  coeparison  matrix. 


*’^****  circle  hcM  veil  these  weights  reflect  your  cpinions. 
Poor  Adequate  Good 
123  4567  89  10 

(if  “poor*  -  please  dxw  in  your  "true*  weiAts  and  review 
matrix  below) 


?  Si 

U  O  OS 

u  o  ' 
<  ec  S 
o.  ua  o 

(/)  <  Ow 


appendix  Bi  Fo11om~up  ThMis  Survsy 


Below  aze  the  priorities  derived  frooi  the  aonpariscsn  matrix. 


Please  verify  the  circled  oorparisans  below.  If  there  is  no 
mistake  (inverted  signs,  etc),  just  check  off  the  circle. 
Otherwise,  %<rite  in  what  you  believe  the  jud^nent  dtould  be. 


Which  factor,  as  It  decreases  In  number  or  complexity, 
contributes  more  to  TAV  maintainability? 

(column  -,  row  +) 


MAINTAINABILITY  " 


ft.  ft. 

&  ^  o 

Q:  ^ 

&  & 


NUMBER  OF  JETS 
NUMBER  OF  ROCKETS 
TPS  TYPE 
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Below  aze  the  priorities  derived  fixam  the  ocsrparison  matrix. 


Please  circle  how  well  these  weights  reflect  your  opinions. 
Poor  Adequate  Good 
123  4567  89  10 

(if  "poor"  -  please  draw  in  your  "true"  weights  and  review 
matrix  below) 


PROPELLA.NTS  >1 


g  w  5 

I  “  I 
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Belohr  aie  the  priori  ties  derived  from  the  csnpariaan  matrix. 


Please  circle 

how  liiftii  tbese  %dei^ts  reflect,  your  opinions • 

Poor 

Adequate  Good 

12  3 

4  5  6  7  8  9  10 

1  (xf  **poor'*  •  pXoase  dnw  in  your  "true**  and  ravloif  I 

1  matrix  below) 

_ 

survivability 


Please  verify  circled  oonpariscns  below.  If  there  is  no 
nistalce  {inverted  signs,  etc) ,  just  disck  off  the  earcJe, 
Otherwise,  write  in  what  you  believe  the  judgment  ^uld  be. 


Which  factor  contributes  more  to  TAV  survivability? 
(column  row  +) 


SURVIVABILITY 


A  ^  § 

i  ^  ^  i 

,Cj  ar 

^  S 

w,  Co  ^  ^  Co 

^  ^  a  ^ 


// 


1  -  Equal  Importance 
3  -  Weak  Importance 
5  -  Moderate  Importance  ^ 
7  -  Strong  Importance 
9  -  Absolute  Importance 


LOITER 

MOBILE  LAUNCH 
BASE  ESCAPE 
DISPERSAL  RANGE 
LASER  HARDNESS_ 
RCS 


ni 

□ 

I 
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Below  aze  the  priorities  derived  frcxR  the  oonparison  matrix. 
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Below  are  tha  prioritiea  derived  from  the  oonfsariacn  matrix. 


Vlaeea  circle 

how  well  these  weights  reflect  your  cpinicna. 

Poor 

Adatjuate  Good 

12  3 

4  S  6  7  8  9  10 

(if  "poor*  -  pleaee  draw  in  your  ’true"  wei^te  and  review  I 

1  nfttrix  below) 

— 

W| 


RISKS 


z 

o 

M 


OS 

H 


Z 

o 


Plaaee  verify  the  circled  aan{>arl8ons  below.  If  there  is  no 
I  odetake  (inverted  eigne,  etc) ,  juetl  check  off  the  circle. 

I  Otherwiee,  write  in  what  you  believe  the  jud^aent  should  be. 


Which  contributes  more  to 
technological  risk?  (column  row  +) 


RISKS  -/// 

<?•  ^ 


PROPULSION 

S'iRUCTURE 


c 


B-a 
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Below  are  the  priorities  derived  frcrt  the  oonpeurison  matrix. 


Please  circle  how  well  these  weights  reflect  your  opinions. 
Poor  Adequate  Good 
123  4567  89  10 

(if  "poor”  -  please  draw  in  your  "true*  weights  and  review 
matrix  below) 


APPENDIX  B:  Follow-up  Thosis  Survey 


BatfM  asB  the  prioritiee  derived  fron  tfia  aaDf>eudecn  matrix. 


PXmm  haw  well  theee  weights  reflect  your  opiniens. 

Poor  Aaequets  Good 
123  4567  8910 

{if  "poor”  -  please  draw  in  your  "true"  weights  and  revia# 
matrix  below) 


1 


TAV  EFFECTIVENESS  ^ 


s 


u  u 
a  a, 


2 

> 


O 

o 

•4 


Please  verify  tl»  circled  ooeparisors  below.  If  there  is  no¬ 
mistake  (inverted  signs,  etc) ,  just)  chet*  off  the  circle. 
Otherwise,  write  in  what  you  beUeve  the  judynent  should  bo. 


Which  element  is  more  important  to  establish  the 
feasibility/effectiveness  of  the  TAV  concept? 
(column  row  ■♦•) 


TAV  EFFECTIVENESS 


REQUIREMENTS/GOALS 

ferfor:ia;ice 

■V  COSTS 

RISKS 
LCGISTICS 
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Feedback  (reasons  whv  people  deviated  ^rom  group  mean) 

Beneral 

11  In  some  cases  your  criteria  'for  comparison  tended  to  compare 

an  absolete  (global  range)  with  variables  and  in  general  categorv  with 
a  sub-element  o-f  that  category. 

11  In  no  cases  were  missions  specified,  which  is  a  driver  in 

weighting  criteria.  I  generally  used  the  Space  Command  missions. 

Reaui rement s/Goal s 

2  If  it  can’t  meet  valid  requi rements/goal s.  it  should  not  be 

done  (but  they  should  be  revised). 

9  Reaui rements  rank  behind  performance  and  survivability,  such 

as:  manned  flight,  global  reach,  sortie  ops.  and  flexible  basing. 

11  Reaui rements.  goals,  and  performance  are  subsets  of  the  same 

element.  The  importance  of  some  factors  under  each  heading  drives  the 
weighting.  Risks  are  a  subset  of  costs.  Survivability,  logistics,  and 
environmental  issues  tend  to  group  together  also. 

13  This  is  not  a  balanced  set  of  reauirements.  Some,  like  rapid 

response  and  flexible  basing  are  auite  important.  Some,  like 
horizontal  TO  and  landing  are  not  very  important.  If  the  vehicle  is 
capable  of  rapid  response,  flexible  basing  and  etc,  it  does  not  matter 
what  the  TO/landing  mode  is. 

4  Goals  moves  down  because:  S  minute  launch  capability,  2 

mi ssi ons/day,  and  20,0001be  payload  take  away  a  lot  of  flexibility  you 
may  need  in  funding  later  on  (STS  isn’t  to  20K  vet.  regardless  what 
NASA/DOD  savs). 

5  2  mi  SSI ons/dav.  single  stage,  few  orbits  are  either  not 
possible  or  undesirable.  For  example,  single  stage  is  undesirable 
because  it  increases  weight  and  degrades  performance. 

Costs 

4  It  is  simply  unreasonable,  and  a  frequent  criticism  of  DOD, 

that  we  do  not  adequately  address  costs.  Today,  as  well  as  tomorrow, 
the  competition  for  resources  should  never  be  taken  lightly  and  such  a 
prgam  like  TAV  will  be  priced  in  the  billion  %  range.  Let's  not  plan 
without  this  in  mind.  Some  may  say  a  high  ranking  here  is  synonymous 
with  its  importance. 

6  There  is  enough  resistance  throughout  the  Air  Force/DOD  to 
kill  this  unless  it  makes  significant  improvements  in  the  payload  to 
orbit  costs  catagory. 

9  Although  costs  are  important,  I  have  problems  in  my  mind 

viewing  costs  as  a  mam  driver.  TAV  will  be  very  expensive.  It  may 
very  well  shelve  TAV  for  a  decade  until  the  dollar  figure  representing 
LCC  isn’t  so  scary,  but  I  believe  that  TAV  will  be  and  cost  will  not 
stand  in  its  way. 

1 
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Ri  sks 

t,  I  knoM  more  about  the  technologies  required  than  last  vear. 

The  large  size  c-f  the  initial  investment  makes  risk  more  important. 

13  The  items  listed  under  risks  are  extremely  important  since 

they  largely  determine  the  -feasibility  ot  the  vehicle  concept. 

Sur Vi vabi 1 i ty 

4  Survivability  seems  somewhat  less  an  issue  in  the  basic  TAV 

concept  with  its  relatively  short  taskings.  Dispersals  should  not 
necessarily  be  a  problem  since  launches  -from  runway  environments  could 
occur  from  a  multitude  of  suitable  launches  (5  min  or  less). 

6  Simply  cannot  be  done  to  same  degree  of  a  JP-4  fueled 

B-1. which  appears  to  the  standard.  However.  I  do  think  some  level  of 
survivability  can  be  achieved  by  operational  decisons  (better 
planning)  versus  system  design. 

7  Survivability  is  a  subset  of  performance.  If  it  can’t  survive, 
it  can’t  perform. 

9  Performance  (payload)  and  the  ability  of  TAV  to  survive  in 

order  to  deliver  a  payload  are  paramount.  Without  these,  there  would 
be  no  TAV. 

Logi sties 

2  Logistics  includes  maintainability  which  is  a  design  element. 

Logistics  also  is  driver  in  LC  costs,  sortie  generation  capability, 
reaction  times,  etc.  It  is  a  crucial  element  to  be  considered  early: 
e.g.  proper  consideration  to  maintainability  aspect  of  design  up  front 
can  facilitate  rapid  low  cost  changes  (mods)  to  survivability  (threats 
change),  performance  (propulsion  technology  changes),  ability  to  cope 
with  environ  (tech  changes),  etc. 

4  Don’t  build  if  you  can’t  properly  support!  Support 

capabilities  often  the  first  to  be  cut  (funding  wise)  when  programs 
are  in  trouble.  If  we  are  going  to  hav  TAV,  let’s  be  able  to  support 
it  so  it  can  do  its  mission  up  front  and  not  let  maintainability 
Buffer  in  the  long  run. 

6  Again,  if  the  Tcch/Req/Cost/Perf  categories  can  be  solved, 
logistics  are  a  tail  and  can  be  solved. 

7  Things  such  as  environmental  impact  and  logistics  are  too  far 
down  the  road  to  worry  about  at  this  point.  The  real  quesiton  at  this 
point  should  be:  "Do  the  requirements  justify  the  costs  and  risks 
associated  to  achieve  the  desired  perf ormanceT* 


ti. 


6-13 


APPENDIX  C3i  AHP  SufliMry  Program 


procedure  DoOncFileCvar  filenue: integer): 

(t  Finds  the  eigenvectors  for  each  tile  and  stores  thee 
in  a  large  eatrix  ‘results’  tor  output 

t) 

var 

crit,i  :  integer; 
begin 
crit  :=  0; 

reset (eatrix Igintileltilenuel); 
rcadln(eatrixl,last): 
readlnleatrixl, author); 

cluster 

readlnlaatrixl.dieenlclusterl); 

repeat 

Mhile  dieenCcluster]  =  0  do  readln(eatrixl.dieenCclusterl); 
Get)1atrjx(dieenCclu5terl):  (»  returns  inputs  v  critenontil  ») 

Fi  ndMeights(dieenCclu$terl); 
tx  i  :=  1  to  dieenlclusterl  do  begin 
crit  :=  crit  +  1; 

ResultsCcrit.tilenuil  :•  uCil; 
end;  (t  tor  t) 

it  criterionlclusterl  <)  last  then  begin 
cluster  !=  cluster  >  1; 
readln(aatn  xl, dieenCcluster  1): 
end;  II  it  I) 

until  criterionCclusterl  ^  last; 

criterionCclusterl  :*  ’bogus’;  (I  prevents  preeature  tereination  tor 

subseouent  tile  nuebers  I) 
MritelnCFile  l’,tilenua:2,’  done.’); 
closeleatrixl); 
end;  (I  OoOneFile  I) 

procedure  DisolaySueearylvar  Liaitiinteger); 

II  displays  judges  1-13  priority  vectors  and  also  coeputes  the 
standard  deviation  I  10  and  the  aritheetic  aean  ot  the  priority 
vectors  and  outputs  the  results. 

II 

var 

tilen, start, cease, crit, i,j  :  integer; 
totci,seanci  ;  real; 

Heandeight  ;  coluan: 

(tmiiiiiiiiiiiiii) 
procedure  FindClusterHeights; 

(I  suborocedure  to  coepute  the  standard  deviation  and  antheetic  nean 
ot  the  vectors.  Could  have  been  accoeolished  Mhile  displaying  the 
eain  part  ot  the  cluster,  but  that  Muld  have  exceeded  the  sue 
tor  an  Apple  Pascal  procedure. 

II 

<ar 

total, SuaSpr  ;  real; 
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prKtdure  FindMeightsInua  :  integer); 

(t  finds  eigenvKtors  as  in  consistency  check  progaa  and  folloes 
the  logic  and  variables  shoMi  in  Saaty’s  Decision  naking  for  Leaders  t) 
var 

c7,a2  :  coluan; 

laada,d,s,t4,  roesue,  atroM  ;  real; 
k,i,j  :  integer; 

begin 
t4  :»  0; 

for  1  1  to  nua  do  begin 

roHsua  :=  0; 
for  j  :=  1  to  nua  do 
roHSua  :=  roasua  *  c[i,j]: 
n2[i]  :=  roasua; 
t4  i-  t4  *  roasua; 
end;  (I  for  i  I) 
for  i  :=  1  to  nua  do 
N2[i]  a2[il/t4; 

k  0; 
repeat 
t4  :=  0; 
k  :=  k+l; 

for  i  1  to  nun  do  begin 
atroa  :*  0: 
for  j  :*  I  to  nua  do 
atroa  s*  atroa  +  cti,jl  •  n2[,]; 
all]  atroa; 

t4  ;»  t4  ♦  atroa; 
end;  (t  for  i  tl 
d  :*  0: 

for  i  1  to  nua  do  begin 
all]  a[i]/t4; 

d  :=  d  ♦  abslaCi]  -  a2Ci]); 
end;  (t  for  i  1) 
for  i  ]  to  nua  do 
<2[|]  ail]; 

until  (k.lOCO,'  or  !d-'1.0E-6); 

'or  1  :=  1  to  nun  do  begin 
s  :=  n; 

*Of  J  :=  !  to  nut  do 

S  :=  S  ♦  Cfl. j]  «  w[j]; 
c7[il  :=  s: 
end:  (I  for  i  I) 

5  0: 

for  1  :=  1  to  nua  do 
s  :=  $  ♦  c7Ci]/a[i]: 

laada  :=  s/nua;  (I  not  used  in  this  oroorat  II 
Cl [cluster, filenual  :=  (latda-nua)/ (nut-1); 
crCcIuster.filenuiJ  :=  ciCcluster, fiienutl/rCnuaJ; 
end:  (I  Finddeights  I) 
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progru  eigenvictar (input. output): 

(t  reads  rau  data  and  coaputes  eigenvKtors,  ci,  and  cr,  and  outputs 
to  tile  'eigout.'  Eigout  includes  the  uans  and  variances 
(10  t  Standard  Deviations)  of  each  eleient  (subobjective)  ot  the 
norulized  eigen  vectors.  That  is,  the  variance  betueen  judges’ 
priority  vectors  and  the  eean  vector.  Note:  the  data  in  this  thesis 
coaputed  the  aean  and  variance  using  judge  13,  which  was  the  geoaean 
input  'judge.*  This  caused  slight  conservative  bias  that  has  since 
been  corrKted  in  this  prograa. 
t) 

uses 

transcend;  (t  necessary  for  Apple  to  do  logarithas  t) 
type 

aatrix  =  arraylO. .9  ,0..9  1  of  real; 
coluan  =  arrayCl..lSl  of  real; 
var 

c  :  aatrix;  (I  AHP  input  coaparison  aatrices  t) 

results  :  arrayCO. .40,0. .151  of  real;  (t  Suaaary  of  Judges  priorities  t) 

ci.cr  :  arraylO.. 10,0. .151  of  real;  (I  ci.crlcluster, judge!  I) 

diien  :  arrayn..lO]  of  integer;  (I  diaension  of  [cluster]  read  in  t) 

H.r  :  coluan;  (t  weights  t) 

filenua.cluster.Liait:  integer;  (I  liait  *  user  input  I  judges  files  I) 

outaatrix.aatrul  :  text;  (t  output,  input  files  I) 

author, last, title  :  string;  (I  lastsflag  criterion  to  stop  I) 

infile, criterion  :  arravC1..13]  of  string;  (I  judges  infile  naaes,  cluster  criterions  I) 

procedure  Getnatrix(nua: integer); 

(I  reads  in  half  aatrix  of  AHP  coaparisons  and  converts  negative  nuabers 
to  the  recriprocal  ratio  value  while  coaoleting  the  aatrix 

t) 

var 

i.j  :  integer; 
begin 

for  1  :=  1  to  nua-1  do 
for  j  :=  i+l  to  nua  do 
read(aatrixI,cCi,j]); 
readlnlaatrixl); 

readlnlaatrixl.criterionlciusterl) ; 
for  1  :=  1  to  nua-1  do  begin 
for  j  ;=  i>l  to  nua  do  begin 
if  cCi.j]  <  0  then 
cli,,]  :=  -l/cCi,j]; 
clj.il  :=  l/c[i,j!; 
end;  (I  for  j  I) 
cli,i]  1; 
end;  (t  for  i  I) 
cCnua.nual  ;=  1; 
end:  (I  procedure  SetHatnx  t> 
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(ttttttitttttttttt) 

procedure  InitNeu; 
bei|in 

infilet  11  'IS: judgclt.text’;  infilel  21  :=  ’15: judge2t.text’; 

infilet  31  ’tS: judgeSl.text’;  infilet  41  ’iS:judge4t.text’; 

infilet  SI  :=  ’IS: judgeSt.text’;  infilet  61  ’45: judge6t.text’; 

infilet  7]  ’IS: judge7t.text’;  infilet  81  'iS: judgeSt.text’; 

infilet  91  ’t5:iudge9t.text’;  infiletiOl  ’tSijudgelOt.text’; 

infiletlll  ’ISijudgellt.text’;  infiletl21  :=  ’fS: judgellt.text’; 

infiletl31  ’ISigeoieent.text’: 
end;  (t  InitOld  tl 

begin  It  initialize  t) 

HriteInl’This  prograe  Hill  take  ran  data  froe  a  tax  of  12’): 

HTitelnl’ judges”  files  and  Hill  coapute  leans  and  ’); 

HritelnI’Hill  output  the  results  on  file  l5:geoaean’): 

Hriteln; 

HTitel’Do  you  nant  to  use  new  (updated)  or  original  data  (n  or  o)’); 
readln(ansHer); 
if  ansuer  =  'o'  then  InitOld 
else  begin 

nriteln  (’NeH  Files  being  used’): 

InitNen; 
end;  It  else  t) 

Hritel’HoM  lany  files  do  you  Hant  to  use?  ’):  readln(ans): 
if  ans  =  1  then  begin  It  note:  ans  passed  globally  t) 

MTitel’Ithich  file  nuiber  do  you  uant  to  Hork?  ’):  readln(filenui); 
infiletn  :=  infiletfilenual;  It  if  1  file,  only  infiletll  read  tl 
end;  it  if  t)  It  note:  filenua  overwitten  in  lain  prograa  t) 
end;  It  initialize  t) 

begin  It  lain  grograa  t) 
reHrite(outiatrix,'IS:geoKan.text'); 

Initialize; 
filenua  1; 

GetFile(filenua): It  nua  of  crit=i,  nua  of  P-H  coap=j,  in  Surveyti.jl  t) 
CopyFile (survey);  It  copies  survey  into  bogusti.jl,  ie  inits  bogus  t) 
while  filenua  <  ans  do  begin  It  go  through  files  in  order  t) 
filenua  :=  filenua 
GetFilelfilenui); 

GatherBogus;  It  hultiolies  Bogus  by  Survey  t) 
end;  It  uhile  t) 

if  filenua  ?  1  then  AverageBogusIfilenual;  It  Takes  'filenua*  root  Bogus  t) 
OisplayBogus; 
close  (outaatrix.lock); 
end.  (I  prograa  t) 
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procedure  OispUyBogut; 

(I  places  the  geoactric  aean  of  the  inputs  back  into  a  sieilar 
file  structure  as  uas  inputed.  The  difference  is  that  the 
output  needs  to  have  'O'  put  in  the  appropriate  spots  inorder 
that  subsequent  prograas  uill  be  able  to  sequence  the  priority 
vectors 
t) 
var 

diaension, crit, count, i,j  :  integer; 
begi* 

HTitelnloutaatrix.last); 

NritelntoutaatriXi’Eeoaetric  Hean  for’,filenua:3,’  Judges’); 
witelndast); 

witelni’Beoaetric  Hean  for’,filenua:3,’  Judges’); 
for  cnt  :=  1  to  9  do  begin 

diaension  :=  roundlsurveyCcrit.Ol);  (I  use  'dieen.*  for  understanding  I) 
uriteln(outaatrix,diaension:3): 

Mriteln(diaension:3); 
count  :=  0; 

for  1  :=  1  to  diaension-l  do  begin 
for  j  i>l  to  diaension  do  begin 
count  :=  count  +1; 
if  bogusCcrit,count]<l  then 

bogusCcrit, count]  *l/bogusCcrit, count]; 

NT i te (outaatr i x , bogus [crit, count ] :8: 4) ; 

Mrtte(bogus[crit,count]:6;4) ; 
end;  (I  for  j  II 
HTitelnioutaatrix): 
vriteln; 

end;  (I  for  i  t) 

«riteln(outaatrix,criterion[crit]): 
witelnicriterionCcrit]); 
end;  (I  for  crit  tl 
end;  (I  procedure  OisplayBogus  t) 

procedure  Initialize; 

II  sets  up  files,  note:  apple  specific  naaes.  Need  to  lable  files 
ever  svstea  calls  thea, 

I) 

var 

answer  :  string; 

(ttittittititittit) 

praedure  InitOld; 
begin 

inf  lie!  1]  :•  ’IS: judgel.text’ ;  infilel  2]  :=  'IS: judgeZ.text’; 

infilel  3]  ;=  ’I5:judge3.text’;  infilel  4]  ;=  ’IS; judge4.text’; 

infilef  5]  ;=  ’ISijudgeS.text’:  infilel  61  ’IS: judgeh.text’; 

infilet  7]  :=  ’IS: judge7.text’ ;  infilel  81  ;=  ’IS; judgeS.text’; 

infilel  9]  :=  ’IS: judqe9.text’;  infilellOl  :=  ’IS; judgelO.text’; 
infilellll  :=  ’IStjudgell.text’;  infilell2]  ;=  ’IS:judgell.text’; 
infilell3]  ’ISigeoaean.text’ ; 
end;  (I  InitOld  I) 
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praedure  CopyFilelvar  survey  :  utrix); 
var 

diaeflsiofl, crit, count, i,j  :  integer; 
begin 

for  crit  :=  1  to  9  do  begin 
dieension  round(survey[crit,0]); 
bogus(crit,0]  round (surveytcrit, 01); 

count  0; 

for  i  1  to  diaension-1  do  begin 

far  j  i'fl  to  dieension  do  begin 
count  count  >1; 

bogustcrit, count]  :=  surveytcrit, count!; 
end;  tt  for  j  t) 
end;  (t  for  i  t) 

end;  (I  for  crit  do  not  use  criterion  because  need  constants  for  aver  t) 
end;  (t  procedure  CopyFile  t) 

procedure  fiatherBogus; 

It  product  of  'n'  factors  and  take  nth  root  later  t) 
var 

dieension, crit, count, i.j  :  inte.ocr: 
begin 

for  crit  1  to  9  do  begin 

dieension  round(surveY[crit,0]);  (I  jUst  for  ease  of  understanding  t) 
count  ;*  0; 

for  1  1  to  dieension*!  do  begin 

for  j  irl  to  dieension  do  begin 
count  :*  count  ♦!; 

DogusCcrit, count]  bogusCcrit, count!  t  survevCcrit, count!: 
end;  (I  for  ]  I) 
end;  It  for  i  II 
end;  (I  for  crit  II 
end;  (I  procedure  Batherbogus  I) 

function  root (base: real;  niinteger)  :  real; 
begin 

root  exp(  ln(abs(base))  /n  ); 
end;  II  root  II 

procedure  AverageBogusIniintegerl; 

(I  'n'  =  nuefile  I) 
tir 

dimension, cnt, count, i,j  ;  integer; 
begin 

for  crit  1  to  9  do  begin 

dieension  :=  raund(bogusCcrit,0}); It  dieension  for  ease  understanding  I) 
count  0; 

for  1  1  to  diaension-l  do  begin 

for  ;  i+1  to  dieension  do  begin 

count  :=  count  ♦!; 

boguslcrit, count]  :=  rootlbogus[crit,count],n): 
end;  II  for  j  I) 
end;  (I  for  i  II 
end;  (I  for  crit  II 
end;  il  procedure  AverageBogus  II 
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oroarai  geoMan (input, output); 

(I  reads  in  rau  judge  data  and  cosbines  tht  judges  inputs  by 
the  geoeetric  eean  and  puts  the  output  to  tile  geoeean 
note:  to  be  read  by  prograe  ‘Consistency(3»eck,*  the  geoaeae 
tile  needs  to  have  seguencing  aeros  as  shoun  in  the  saaple 
input  tile 
t) 


uses 

transcend; 

type 

eatrix  »  arrayt0..9,0..361  of  real; 


var 

bogus, survey  :  eatrix; 
criterion:  arrav[0..91  of  string; 
infile:  array[0..13}  of  string; 
filenue,nuefiles,ans, count  :  integer; 
auteatrix.aatrixl  :  text; 
author, last  :  string; 


praedure  BetFiledilenue  :  integer); 

(t  ‘tilenue*  is  the  file  sequence  nueber.  This  procedure  puts  entire 
file  into  a  9  1  36  array  called  survey.  The  i  1  0  eleeent  is 
the  diaension  of  the  ith  (of  9)  criterion  t) 
var 

dioension.crit, count, i,j  :  integer; 
begin 

resetlaatrixl.infileCfilenual); 

readln(satrixl,last);  (t  last  -  subtitle  of  last  eatrix  to  enter  t) 
readlnlaatrixl.author); 
readlnleatrixl, diaension); 
for  crit  :=  1  to  9  do  begin 
while  dieension  =  0  do  readln(aatrixl,dieension): 
survevlcritiOl  :=  diaension; 
count  :=  0; 

for  1  :=  1  to  diaension-i  do  begin 
for  j  i+1  to  diaension  do  begin 
(I  i,j  count  variables  to  Iteep  eatrix  foraat  straight  t) 
count  :=  count  *1: 
readleatrixl,  survey! crit, count 1 ) : 
if  surveyCcrit.countl  <  0  then 
surveyCcnt. count]  :=  -l/surveylcrit.countl; 
end;  11  for  j  I) 
end:  !t  for  i  I) 

readln  eatrixl);  (f  reset  pointer  to  start  of  line  t) 
readln  (eatrixlicriterionlcritl); 
if  criterionlcrit]  O  last  then  readln laatriil, diaension) ; 
end;  (I  for  crit  do  not  use  criterion  because  need  constants  for  aver  I) 
closelaatnxP; 

end:  (1  procedure  Set^ile  t) 
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for  i  :=  I  to  n3  do  begin 

count  count  *  1;  (t  count  is  total  t  in  current  generation  t) 
coeposCcount]  b[elee,il;  (t  b  needed  lor  display  t) 
end;  (I  lor  i  f) 

totrandoa  :=  totrandon  *  prevcoapCeleel  t  r[n3]; 
totconsis  totconsis  *  prevcoepleleel  t  ci; 
end  It  il  n3  >  1  t) 

else  begin  It  only  one  eleeent  related  t) 
count  :=  count  ♦  1; 
b(elee,l]  :=  prevcoaplelsal; 
conposlcountl  :=  prevcoipleleel; 
end;  it  else  t) 

end;  (I  il  last  -  allows  putting  last  anywhere  I) 
end;  (t  lor  elea  t) 
end;  (I  NextLevel  tl 

begin  (t  progran  saaty  I) 

reset(eatrixl,’l5!Mtri«l.text’);  (I  lirst  generation  prograe  -  see  prograe 

eigenvector  lor  better  eethod  to  read 
in  files  t) 

rewr i te (outaatr i x , ' IS : outaatr i x . tex t ’ I ; 

InitR; 

L  J*  2; 
elei  :=  0; 

totconsis  :=  0;  totrandon  :-0; 

readlnleatrixMast):  Il  last  ^  subtitle  ol  last  oatrix  to  enter  t) 

readlninatrixl, title); 

readlnleatrul.nl); 

BetHatrixInl);  (I  returns  subtitle:  flag  to  continue  to  next  level  t) 
FindHeightsInl); 

displayinl):  (I  aakes  hard  copy  ol  orginal, since  1*2,  btl  not  necessary  I) 
if  cr  >  O.IO  then  begin 
consistencylnl); 

displaylnl);  (t  final  hard  copy  after  consistency  revisions  t) 
end;  It  if  cr  t) 
for  1  :*  1  to  nl  do  begin 

orevwtli]  :*  wCil;  (t  can’t  reaeeber  what  orevwtll  is  used  for  t) 
orevcoapCil  ;=  orevwtCiI; 
end;  (t  for  i  I) 
while  subtitle  '>  last  do  begin 
L  :=  L  +  1; 
for  1  :=  I  to  nl  do 
for  j  :*  1  to  nl  do 
bCi.jl  :=  0; 

Nextlevel;  (I  returns  count,  coeposlil,  and  subtitle-for  while  looo  I) 
for  1  :*  1  to  count  do  prevcoapCil  :=  coiposCil: 
nl  :*  count; 

end;  it  while  subtitle  last  goto  next  level  t) 
writelnloutaatrix); 

writelnloutaatrix.’The  Consistency  of  this  hierarchy  =  ’,totconsi5/totrandoi:i:4) ; 
writelnCThe  Consistency  of  this  hierarchy  =  ’, totconsis, 'totrandoa:6:4); 
close'.outaatrix.loclt) ; 
ctoselaatrixl; ; 
end.  (1  prograa  saaty  t) 
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ior  i  :*  1  to  si  do  begin 
for  j  1  to  si  do 
write  (cli,j]:b:2,’  ’)» 
Nrite(N[i]:6:5i’  ’); 
if  L  >  2  then  iirite(b[elee,i]:7M); 
MTiteln; 

end;  (t  for  i  t) 

tritel’Uidaleax):  Maida:b:4); 
writeC  C.I.:  ’,ci:6:S); 
witelnC  C.R.:  ’,cr!7:5); 
end;  (t  display  tl 


procedure  InitR; 

(t  Randoe  Consistency  Table  t) 

begin 

rtt]  :=  0.0;  rC2I  :=  0.0  ;  rC3]  ;=  0.58; 
r[4]  :=  0.9;  rCS]:=1.12;  rC61  ;=  1.24; 
rm  :=  1.32;  rlSl  :=  1.41;  r[9]  :=  1.45; 
rtl0];=  1.49;  r[ll]:=  1.51;  rC121;=  1.48; 
r[13]:s  1.5b;  r[14]:=  1.57;  r[15]!*  1.59; 
for  i  :*  1  to  33  do  factorlil  ;=  ’bogus’; 
end;  (I  InitR  tl 

procedure  NextLevel; 

(t  nl  IS  used  as  the  nunber  of  ’parents’  in  the 
current  level.  Variable  naaes  froe  Saaty. 

t) 

var 

i,  j  ;  integer: 

59  :  real; 
begin 

count  :=  0;  (t  left  to  right  nueber  of  siblings  tl 

for  elee  :=  1  to  nl  do  begin 
if  subtitle  <>  last  then  begin 

readln(utrixt,n3l:  (t  n3  =  I  of  siblings  of  parent  ’eleii’  II 
if  n3  /  1  then  begin 
GetNatrix (n3l; 

FindWeights(n3l; 

for  1  :=  1  to  n3  do  (t  b  necessary  for  display  tl 
blelea.il  :=  wCil  t  prevcoapCeleil;  (t  this  does  not  work  for 
•fevel  greater  than  3.  The  fix  would  cost  ae  a  day  I 
don’t  have,  sorry.  I  orefer  prograe  ’eigenvector"  as 
a  better  prograa  to  copy,  anyway.  This  was  ay  first 
orograa,  and  it  is  a  little  rough  and  hard  to  follow. 
However,  if  vou  are  reading  this,  I  aa  preaching  to 
the  choir,  tl 
Display  (n3l ; 
if  cr  >  a.  10  then  begin 

consistencyinjl ; (t  unfortuatelv,  pera  copy  of  changes  only  I! 
for  1  :=  1  to  n3  do  blelea.il  :=  wCil  t  prevcoaplelea]; 
Display ln3l;  (t  final  hard  copy  with  revised  b  wts  tl 
end;  (t  if  cr  I) 
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Mrite  (’  Colun:’);  read(bigcol) ; 
writiln; 

end;  (t  if  eant  to  change  location  t) 
irite  (’Nrite  value  you  eant  to  insert:  ’): 
readlnlclbigroHfbigcol ] ) ; 

NTiteln; 

cCbigcol,bigroH]  :=  * 'c[bigroN,bigcol}; 

FindNeights(sl); 

NTiteln  (outeatrix,’  actual:  [’.bigroN:2,’,’,bigcol:2,’]  =’, 
cCbigroN,bigcol]:3:l,’  uith  C.R.=’,cr:7:S); 

ScreenOi  splay  (si); 
if  cr  >  0.10  then  consistency (si); 
end;  (t  if  aos  t) 
end;  (t  procedure  consistency  t) 

procedure  display(sl:integer); 
var 

i,j  :  integer; 
begin 

oldbigcol  :s  0;  oldbigrou  0;  (t  for  consistency  called  in  display  t) 
Nriteln(outeatru); 

if  L  *  2  then  uriteln  (outeatrix, title;20,’  RESULTS’); 
Nriteln(outaatrii,subtitle:20,’,  Factor  #’,elee:2,’  Level  l’.L::)i 
Nriteln(outaatrix); 

Nrite(outeatrix,’r:4,’  ’); 
for  1  :s  2  to  si  do 
write (outeatrix,;: 6,'  ’); 

Nrite(outeatrix, ’  Height’); 
if  L>2  then  writeln (outeatrix,’  Coep-Ht’)  else 
Nriteln(outaatrix); 
for  1  :s  1  to  si  do  begin 
for  j  ;=  1  to  si  do 
write  (outeatrix, c[i,jh6:2,’  ’); 

Hrite(outBatrix,N[i]:&:5,’  ’); 

if  L  2  then  Nrite(outeatrix,b[elee,i]:7:4); 

Nriteln(outaatrix); 
end;  (4  for  i  t) 

Nrite(outaatrix,'landa(aax):  ’,laada:&:4) ; 

•ritefouteatrix,’  C.I.:  ’,ci:6:5); 

NTitelnlouteatrix,’  C.R.:  ’ .ci/r[sn:7:5); 
if  L  )  2  then  begin 
writeln; 

*riteln(5ubtitle;20,’.  Factor  f’.elee:2,’  Level  r,L:2); 
writeln; 

end:  !l  L  <>  2  accoeplished  so  that  subtitles  print  out  right!) 
write(’r:4,’  ’); 
for  1  2  to  si  do 

write(i:6,’  ’); 
write!’  Height’); 
if  L)2  then  writeln!'  Coep-Ht’)  else 
writeln: 
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l>2  th«  iritelnC  C(Mp-W’)  elst 
Mritcln; 

for  i  1  to  si  do  begin 
for  j  ;=  1  to  si  do 
MTite  (cli, j]:6:2,’ 

Mrite(Mlil:6:S,' 

if  L  >  2  then  irite(bteleo,i];7:4); 

(t  note;  bCl  gets  overeritte:  for  L  >  3  so  not  valid  at  L  >  3  I) 

MTiteln; 

end;  (t  for  i  t) 

MriteClaadalaax):  ’,laeda;6:4); 
uriteC  C.I.;  ’,ci:6:5l; 
witelnr  C.R.:  ’,cr:7:5); 
end;  (I  subprocedure  ScreenDisplay  <) 

procedure  consistency (si: integer); 

(I  calculates  eatrix  eleeentlbigroe.bigcoll  that  lost 
contributes  to  the  inconsistency.  Alloes  user  the 
option  of  changing  the  location  and  eagnitude  of 
input, 
t) 
var 

dtfold.difneH  :  real; 
begin 

difold  ;=  0: 

for  i  1  to  st  do  begin 
for  j  1  to  si  do  begin 
if  (i<>  oldbigcol)  or  (jOoldbigroe  )  then  begin 
difnee  cCi, j^tHCjl/Nlil; 

(difnew  ;*  abslcCi,  jl-eCil/nCjl);} 
if  difnee  >  difold  then  begin 
bigrou  i=  i: 
bigcol  jj 

difold  :=  difnen;  (I  only  updates  if  greater  I) 
end;  it  if  difnen  t) 
end;  (f  if  i<>  *> 
end;  (I  for  }  t) 

end;  !t  far  i  I) 
cldbigrj*  :=  bigrou: 
oldbigcol  1-  bigcol! 

write  ('Ele«ent[’,bigron:2.’.’.bigcol:2,’]  should  bei’, 

t*[bigrot*]/w[bigcol]:5:2.’.  Tvoe  ’’v”  if  vou  want  to 
’reolace  it:'>: 

ante  (outaatrix.’(’.bigroa:2,’,’,biqcol:2,’)  should  be:’, 
Mlbigroal/albigcol  ]:j:2); 
readln(ans); 
witeln; 

if  (ans  =  ’v’)  or  lans  =  ’V")  then  begin 
ante  (’Do  you  aant  to  out  value  in  another  locition  (v  or  n)'”): 
readlnlans): 

if  lans  ~  ’v’)  or  (ans  =  T)  then  begin 
write  (’  Roa:’);read  (bioroa); 
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begin 
t4  ;*  0; 

for  i  :=  1  to  nue  do  begin 
roHSUi  :=  0; 
for  j  :=  1  to  nue  do 
roNsue  :=  rousue  *  c[i,j]; 

H2(i]  roMSua; 
t4  :=  t4  ^  roMSue; 
end;  (I  for  i  t) 
for  1  :=  1  to  nue  do 
m2[i1  M2[i}/t4; 

k  ;=  0; 
repeat 
t4  ;=  0; 
k  :=  k+1; 

for  1  1  to  nue  do  begin 

wtroM  :=  0; 
for  j  ;=  1  to  nui  do 
■tro«  ;s  etroe  ♦  c[i,j]  I  e2[j]; 
n(i]  wtroN; 

t4  t4  ♦  utrou; 
end;  (t  for  i  t) 
d  :=  0; 

for  i  1  to  nue  do  begin 

NfiJ  NCi]/t4; 

d  d  -r  absfuli]  -  n2[il): 
end;  (I  for  i  t) 
for  i  :=  1  to  nue  do 
i*2ri3  :=  util! 

until  (k>l000)  or  (d<1.0E-6); 
for  1  :*  1  to  nue  do  begin 
s  :=  0: 

for  j  1  to  nue  do 

5  :=  s  ♦  cCi.j]  I  wCjl; 
c7[i]  ;=  s; 
end;  (I  for  i  t) 

i  :=  0; 

for  1  :=  1  to  nue  do 
5  :=  5  +  c7[i]/e[i]; 
iMda  :=  s/nue; 

Cl  :=  (laeda-nue!/(nue-l); 
cr  :=  ci/r[nui3; 
end;  <t  FindWeights  I) 

procedure  ScreenDi  spl  ay  1  si  .‘integer);  (t  consistency  too  long  4) 
begin 

Mritelnlsubtitle;20,’,  Factor  •’,elea:2,’  Level  t’,L:2); 
uriteln; 

v<rite(’r:4.’  ’); 
for  1  :=  2  to  si  do 
wite(i:6,’  ’); 
uriteC  Neiqht’); 
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progrm  ConsiitencyCheckd nput,autButl ; 

(I  Reads  inpet  file  ’wtrixl'  of  AHP  paireise  coeparisons. 

Coaputes  the  eatrix  ci.cr,  and  eigenvectors  after  allOHing 
the  user  to  input  corrections  to  the  aost  "deviate*  inputs.  The 
erroneous  inputs  are  identified  by  the  largest  divergent  ratiOi  as 
explained  in  Saaty  AHP,  page  65.  dote:  this  prograa  is  rough  and 
non-eloquent.  It  coaputes  a  total  hierarchy  consistency  -  the  only 
addition  over  the  ‘eigenvector*  prograa  listed  next, 
t) 

type 

aatrix  =  ar ray [0.. 40,0.. 40]  of  real; 
coluan  =  array[1..40]  of  real; 
var 

c,b  :  aatrix: 

prevHt,a,prevcoap,coapos  :  coluan; 

r  :  array[1..13]  of  real; 

totconsis,totrandoa,ci,cr,laada  :  real; 

oldbigroa,oldhigcol, count, «3,i,j,L,elea,nl  :  integer; 

bigro«,bigcol  :  integer; 

subnaaes,outcoap,outaatrii,aatrixl  :  text; 

last, title, subtitle  :  string; 

factor  :  array  Cl.. 351  of  string; 

ans  :  char; 

atend  :  boolean; 

procedure  SetNatrixtsl: integer); 

(t  Reads  in  AHP  coaparisons  and  inverts  nuabers  <  0  to  coaplete  aatrix  t) 
var 

i,j  :  integer; 
begin 

for  i  1  to  sl-l  do 
for  j  :=  1*1  to  si  do 
readfaatnxl.cCi.  jll; 
readln(aatrixl); 
readln  (aatrix  1,  subtitle); 
for  1  :=  1  to  sl-l  do  begin 
for  j  :=  1+1  to  si  do  begin 
it  cCi.j]  <  0  then 
cli.j]  :=  -l/cCi,jl; 
c[j,iJ  !-  l/cCi.jl; 
end;  (I  for  j  I) 
cCi,i]  :•  1; 
end;  (I  tor  i  I) 
cCsl.sll  1; 

end;  (t  procedure  OetRatrix  t) 

procedure  FindNeightsinua  :  integer); 

It  variable  nanes  froa  Saaty’s  text,  little  intuitive  aeaning  t) 
var 

c7.i»2  :  arravC1..10)  of  real; 
d,s,t4,  roMsua,  utroN  :  real; 
x.i. )  :  integer; 
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APPENDIX  Bs  FolloM— up  Thesis  Survey 


Optional  Detailed  Feedback  (comparison  matrices) 


0  Mai ntai nati 1 itv  is  a  design  function  which  encompasses 

accessabi 1 i ty,  reliability,  modularity,  etc.  Early  concept  formulation 
absoul  tel  y  must  gi  v^e  great  consideration  to  this  critical  design 
el ement. 

11  For  maneuverability  to  be  meaningful  for  mission  applications, 

maneuverability  in  space  is  the  driver.  Maneuverability  for 
launch/recovery  provides  flexibility  and  allowance  for  error.  Orbital 
injectian/selection  is  a  different  issue  from  maneuverability. 

11  Payload  mass  and  volume  are  determined  by  mission  which  drives 

reouirement  for  maneuverability  (orbital  change).  A  higher  orbit  can 
be  traded  for  payload,  etc.  Basically  you  want  to  maximize  all  three 
factors.  Trades  are  based  on  missions  and  cost. 

2  Propellants  are  a  consumable  resource  (supply  problem). 

11  Cryogenic  fits  under  the  category  of  amount  and  different 

types.  Another  case  of  the  total  set  being  more  important  than  the 
subset.  The  terms  cryo  and  storable  both  cover  a  multitude  of  sins. 
Hvdrozine  is  probably  worse  than  LDX  but  LOF  is  probably  worse  than 
both.  The  issue  of  number  of  different  types  is  that  a  failure  to 
provide  one  may  be  a  cause  for  abort  thus  there  is  a  potential  for 
lots  of  small  critical  points  vice  one  or  two  big  ones. 

2  Facilities,  by  your  definition,  may  be  either  "nice  to  have" 

or  "critical." 

11  What  IS  an  element?  Big  rockets,  little  rockets,  expendable 

rockets? 

6  Structure  and  Mass  Fraction  are  not  mutually  exclusive. 

6  Without  global  range,  forget  mission. 

11  The  issue  "global  range"  should  be  expendables  (fully 

reusable)  or  flexible  basing  as  defined  by  Battel le,  not  global  range. 
All  Phase  I  concepts  achieved  orbital  velocity,  even  for  "once  around" 
mi ssions. 

9  Base  escape  and  mobile  launch  essential  (regards  to 

survi vabi 1 itv) . 

11  Invalid  Questions  tend  toward  invalid  analysis.  Analyzing  "How 

high  is  a  duck?"  can  lead  to  interesting  conclusions  but  not 
necessarily  a  MS  degree.  1  wish  you  success  in  your  endeavor. 
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APPENDIX  C3!  AHP  Summary  Program 


begin 
i  :»  0; 

for  crit  start  to  cease  do  begin 
i  :*  i  +  1; 

SuaSqr  :=  0; 
total  :=  0; 

for  filen  :=  1  to  Lieit  do  begin 
total  total  *■  Results[crit,filM],‘ 

SuaSqr  SuaSqr  *  sqr(Results[crit,filen]): 
end;  It  for  filen  t) 

Resultslcritfltl  :=  total/Liait; 

IleanHeightli]  :=  ResultsCcrit,141; 

(t  Insures  NeanVote  in  colunn  foraat  for  CalcZeta  t) 
Results[crit,lS]  :=  sqrtlSuiSqr/Liait-sqrfHeinHeightlil) )tl0; 

(t  Standard  Deviation  is  a  very  saall  nuaber  so  t  10  t) 
end;  (t  for  crit  t) 

end;  It  procedure  FindClusterMeights  t) 

procedure  DisplayflainPart; 

It  displays  the  eigenvalues  for  all  the  judges  t) 
begin 

M'ite(outaatrix,’r;4); 

for  i  2  to  Liait  do  Hrite(outaatrix.i;&>;  Mriteln(outaatrix); 
arite(’l’:4  ); 

for  i  2  to  Liait  do  Mrite(i:6);  ariteln; 
for  crit  :=  start  to  cease  do  begin 
for  filen  :=  1  to  Liait  do  begin 
»rite(ouleatrix,ResLlts[crit,filen]:5:3l; 
tcite(Re5ults[crit,filen]!5;3); 
end;  (t  for  filen  t) 
nriteln;  uritelnloutaatru) ; 
end;  It  for  crit  II 
totci  :=  0; 

fx  filen  1  to  Liait  do  begin 
totci  :=  totci  +  cilclusterjfilenl; 
writeloutaatrix.  Cl  [cluster,  filenhS:  3); 
uritelci [cluster, filen 1:5: 3); 
end;  (I  fx  t) 
aeanci  :=  totci /Li ait; 

HTiteln;  writelnlouteatrix); 
for  filen  :=  1  to  Liait  do  begin 
an  te(outaatrix,cr  [cluster,  filen]:5: 3); 
tr  i  te(cr  [cluster.  filen]:5;  3); 
end;  (I  fx  I) 

HTiteln:  Nritelnloutaatrixl; 
end;  (t  DisplayllainPart  I) 

(ttttttitttttll) 

procedure  DisplayCluster; 

(I  Results["crit'xion,  1..13]  =  judges,  14=aean,lS=Dev  t  10 
calls  OisplayNainPart  and  then  displays  the  vector  aean 
and  Standard  Deviations  of  the  priorities 


appendix  C3s  AHP  Sumnary  Program 


var 

totci,  totcr  :  real; 
begin 

DisplayflainPart; 

•riteln<outeatrix,’Hean’:5,’  Dev  I  10*:11); 

MriteU('Nean’:S,’  Dev  t  10’: II); 
for  crit  start  to  cease  do  begin 
irit«(out«atrix,Results[crit, 141:5:3);  (t  Vector  Hean  for  judges  t) 
Mriteln(outeatrix,ResultsCcrit, 151:8:2);  (t  Dev  t  lO  for  judges  t) 
NTitelResultslcrit, 141:5:3); 
iritelnlResultslcrit,  151:8:2); 
end;  (t  for  crit  t) 
witeln(outaatrix,aeanci:6:3  ); 

Hriteln(aeanci:&:3); 

witeln(outiatrix,ieanci/rtdiien[cluster)l:A:3); 

Hriteln (aeanci/rCdiaenlclusterl 1:6:3) : 

witelnC - >j30); 

end;  (t  procedure  OisplayCluster  t) 

(tttititmtttttmt) 

begin  (t  OisplaySuaeary  t) 

if  cnterionCclusterl  »  ’bogus’  then  criterionlcluster ]  :=  last; 

(I  insures  the  previous  file  does  not  preeaturely  tereinate  this 
prograe  froe  reading  the  last  cluster  (eatrix=eaneuverability  I) 
vriteln(outeatrix,lieit;3,’  Decision  Haters’); 
witelnt’Voting  Sueeary’:20,'  for’,lieit:3,’  Decision  Haters ’); 
start  :*  1;  (I  Begins  neu  cluster  in  ’for"  loop  I) 
for  cluster  :*  I  to  9  do  begin 
cease  :-  start  *  dieenCcluster]  -  1; 
witelniouteatrix,  criterionlcluster  1); 
Mriteln(outaatrix,diaenCcluster}:3); 
uriteln(criteriQn[clusterl,dteenCclu5ter]:3); 

FindClusterMeights;  «  returns  aritheetic  eeans  i  variances  I) 
OisplayCluster; 

start  :=  cease  +  1;  (t  avoids  printing  sase  line  tuice  I) 
end;  (I  for  cluster  I) 
end;  (t  prxedure  DisplaySuaaary  t) 

procedure  Initialize; 

(*  sets  up  files,  note:  apple  specific  nates.  Need  to  label  files 
what  ever  systea  calls  thea, 

I' 

var 

answer  :  string; 

(mmtttttttmt) 

procedure  InitOld; 
begin 

infilet  11  :=  ’ISijudgel.text’;  infilet  21  :=  ’#5; ,udge2.text’ ; 

infilet  3)  :=  ’15; judge!. text’;  infilet  41  :=  'ISijudgei.texf; 

infilet  51  :=  ’ISi.udgeS.text’;  infilet  61  :=  ’15: ,udge6.text’; 

infilet  71  :=  ’15: judge?. text’:  infilet  81  :=  ’I5:,udge8.text’: 

infilet  91  :=  ’l5:)Udqe9.text’;  infiletlOl  :=  ’15: judgelO.text’ ; 
infiletlll  :=  ’15: judgell.text’;  infilell21  :=  ’l5:,iudgeU.text’; 


APPENDIX  C3:  AHP  SuiMnary  Program 


infilen3]  :=  ’ISigeoKan.teit’; 
enO  (I  InitOld  I) 

(ttmtttttitttitt) 

procedure  InitNeu; 
begin 

infilci  n  ’•S:judgelt.text’; 
infilel  31  ’IS: judge3t.text’; 
in^ilei  5]  ’IS: judgeSt.text'; 
infilei  71  :=  ’15: judqe7l.text’; 
infilel  91  s-  'K:judge9t.text’; 
infiletlll  ’IS: judgelit.text’ 

infile[13]  ’IS:geoMenl.text’ 

end;  (t  InitOld  tl 


inlilet  21  :=  ’15; judge2l.text’ 
infiiet  41  :=  'IS: jud9e4l.text’ 
infilet  41  ;=  ’15: judgebl.text’i 
infilet  81  :=  ’#5; judge8l.tex t’ 

infiletlOl  :=  ’15: judgelOl.text 
infiletl2]  ’IS: judgelit.text 


begin  (t  initialwe  t) 

(t  Randoe  Consistency  Table  (reduced  for  problee  )  t) 
nil  :=  0.58: 

rt41  :=  0.9;  r[5]  :=  1.12;  rtil  ;=  1.24; 
rt7]  :=  1.32;  r(81  :=  1.41;  rC9]  :=  1.45; 
rtlOl:*  1.49; 

Nritelni’This  prograe  uill  take  rau  data  froe  a  eax  of  13’); 

NT iteln(’ judges”  files  (13=geoeea(i)  and  coepute  eigenvectors  +  leans’); 
eritelni’and  Mill  output  the  results  on  file  IS:eigout’): 

HTiteln; 

Hrite(’Do  you  lant  to  use  nei  (updated)  or  original  data  (n  or  o)’); 
readlnlans); 

if  ans  -  'o’  then  InitOld 
else  begin 

HTiteln  (’Nen  Files  being  used’); 

InitNev; 
end;  (t  else  t) 

Nritel’HoH  lany  files  do  you  nant  to  use?  ’);  readln(liiit); 
if  lint  :  1  then  begin 

tritel’Nhich  file  nueber  do  you  nant  to  Hork?  ’);  readln(filenui); 
infiletn  :=  infiletfilenuil;  (I  if  1  file,  only  infiletl)  read  t) 
end;  (t  if  t) 
end;  (t  initialize  t) 


begin  It  aain  prograa  t) 
reuriteloutaatrix, ’ ISieigout. text ’ ) ; 
initialize;  It  file  titles  set  and  user  inputs  *liiit'  I) 
for  filenua  1  to  lint  do 
DcOneFilelfilenui); 

DisplaySunaryllint); 
close  (outiatrix.lock); 
end.  (I  prograa  eigenvector  t) 
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APPENDIX  C4:  Vote  Convaraion  Program 


aroqrat  zcta  (input, output); 

(t  Rtads  in  rau  judfos’  files  sequentially  and  outputs  the 

converted  Heights  and  zeta  coefficients  to  file  indvotes.  First 
converts  the  AHP  eatrix  uith  1-9  scale  into  a  voting  eatrix  (0, 1/2,1). 
Calculates  the  aax  nuaber  of  circular  triads  and  subsequently 
the  Kendall  coefficient  of  consistency  (zeta),  follouing  the  procedures 
outliied  by  Kendall  (1970:146).  Tied  ranlis  are  taken  care  of  by 
deteraining  a  range  of  triads  possible  and  using  the  average  to 
derive  zeta  -  see  Dobbins  (1980:30). 
t) 

uses 

transcend;  (I  apple  for  square  root  function  t) 
type 

Htrix  -  array[0..11,0..il]  of  real; 
coluan  3  arravCO.,12]  of  real; 
var 

copy,  VoteCount:  arraytO. .40,0..1S]  of  real; 
c  :  aatrii; 

Zetall.ZetaC  :  array[0..10,0..1S]  of  real; 
ordered, RouVotes, a  :  coluan; 
results  :  arraytO. .10,0.. 10,0. .151  of  real; 
sequence, diaen  :  arraytO.. Ill  of  integer; 
liait,crit,filenua,i,j, cluster  :  integer; 
criterion, infile  :  array(1..14]  of  string; 

I  tea  :  arrayll..41]  of  string; 

AveZeta,CoaZetaU,first,saxH  :  real; 
author,  last  :  string; 
outaatrii,aatrul  :  text; 

NantCopy  :  boolean; 

prKcdure  Naaeit; 

<t  file  of  subob]8Ctive  naaes  I) 
var 

naaes  :  text; 
i.j  :  integer; 
begin 

reset  (naaes, ’ISinaaes. text’ ); 
for  1  :s  1  to  41  do 
readln(nases,iteo(i]); 
close (naaes); 

end;  (t  procedure  naaeit  II 

function  aaxtriads(nua: integer)  :  real; 

(t  deteraines  the  aaxiaua  nuaber  of  triads  possible  for  a 
diaension  of  a  pairuise  coaparison  aatrix 

ttumuuntmtt) 

function  nuaeven  (nua: integer)  ;  boolean; 
begin 

if  trunc(nua/2)  =  nua/2  then 
nuaeven  true 
else 
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nuMvm  false; 
end;  (t  function  nuneven  t) 

(tttttttttttttttltttt) 
bcfin  it  function  naxtriads  t) 
if  nuiivm(nun)  then 
naxtriads  (nun  t  (sqrCnun)  -4))/24 

else 

naxtriads  (nun  t  (sqr(nunl  ■*l))/24; 

end;  (t  function  naxtriads  t) 

procedure  sort(var  colun  :  coluen;  nun  :  integer); 

(t  detereines  seguenct  and  naxiaus  elenent  of  vKtor  I) 
var 

i  :  integer; 
sorted  :  boolean; 
tenp  :  real; 
begin 

(t  pass  through  colue  until  sorted  t) 
repeat 

(t  assune  colun  is  sorted  t) 
sorted  :=  true; 

(t  start  neu  pass  t) 
for  i  ;=  1  to  nun-1  do 
(t  start  neu  pass  t) 
if  colunCil  <  coluali^ll  then  begin 
(}  out  of  order  -  exchange  and  reset  sorted  I) 
teap  ::  colunCitll; 
colunlitll  :«  coluali  1; 
colueCi  ]  teep; 
sorted  t-  false; 
end;  (1  if  colun  tl 

(I  test  sorted  at  conpletion  of  pass  t) 
until  sorted; 
saxK  :=  colusCll; 
end:  (t  sort  t) 

xocedure  BetHatrix (nun: integer); 

(t  reads  in  rau  AHP  coeparisons  and  converts  to  •voting’  eatrix  I) 
var 

i,j  :  integer; 
begin 

for  1  :=  1  to  nui-1  do 
for  i  :=  ir]  to  nus  do 
readleatrixl.cCi,]]); 
readln(aatriil); 

readln(eatrixl,criterion(cluster]); 
for  1  :=  1  to  nue-1  do  begin 
for  j  irl  to  nun  do  begin 
if  cCi,j]  <  0  then  begin 
cli.jl  :=  0: 

ctjjil  ;*  1; 
end  it  If  1/ 
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tlst  if  c[i,j]  *  1  then  begin 
c[i,jl  O.S; 
c[j,i]  :=  0.5; 
end  (t  else  if  t) 
else  if  c[i,j]  >  1  then  begin 
c[i,jJ  :=  1; 
cljjil  :=  0; 
end:  (I  else  if  t) 
end;  (t  for  j  t) 
cli.il  :=  0; 
end;  (t  for  i  I) 
c[nue,nue]  s-  0; 
end;  (t  procedure  GetHatrix  t) 

procedure  FindHeights(nue:integer); 

(t  returns  ulil  :  percentage  of  RouVotes  for  eleuent  i 
also  returns  first:  'first*  part  of  zeta  calculation 
and  zeta:  Kendall’s  eeasure  of  consistence, 
t) 
var 

0,2eta,Su«D,TotalVotes  :  real: 
i,j  :  integer: 
begin 

Total Votes  :=  0; 
for  i  :•  1  to  nue  do  begin 
RoNVotesCil  :»  0; 

(or  j  1  to  nue  do 
RoNVoteslil  RoHVotesCi]  *  c[i,j}: 

Total Votes  :•  TotalVotes  *  RouVotesli}; 
end:  (I  (or  i  t) 

SuaO  :=  0; 

for  I  ::  1  to  nua  do  begin 
crit  :=  crit  ♦  1: 
uCil  :=  RouVotesli 1/Total  Votes; 

SuaO  SuaO  +  RouVotesli II (RouVotesli 1-1); 
VoteCountlcrit.filenual  :=  RouVotesli!: 
end; 

first  :=  nuat(nua-l)t(nua-2)/&:  (t  first  is  used  in  CalcZeta  i) 
0  :=  first  -  SusO/2:  (I  Actual  I  triads  t) 

Zeta  :=  1  -  0/aaxtriadsinua); 

ZetaUlcluster.fi lenual  :=  zeta; 
end:  (I  procedure  FindWeights  t) 

procedure  Di solavVotes (nua: i nteger ) ; 

(t  'cluster', 'RouVotes*.  and  'criterion*  are  passed  globally  t) 
var 

i.j  :  integer: 
begin 

uriteln(criterionlclusterl); 
uriteC  ’:3.’r); 
for  1  2  to  nua  do 
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iriteln(’  Total’:?,’  Heights’); 

4or  i  :=  1  to  nua  do  begin 
for  j  1  to  nua  do 
Nrite(cCi,jl:6:2); 
w i tel n (RoeVotesl i 1 : 6 : 2 , h[ i ] : 8: 3) ; 
end;  it  lor  i  t) 

uritelnCZetaiu)  =  ’,2etallCcluster,filenuel:6:4,’  2eta(c)  =  ’, 
ZetaCCcluster,filenua]:6!4); 

iritelni’ - ’:30)! 

Hriteln(outaatrii,criterion[cluster]); 

NTitetouteatrix,’  ’:3,’l’); 

Tor  1  :=  2  to  nua  do 
Hrite(outaatrix,i:6); 
uritelnioutiatrix,’  Total’:?,’  Heights’); 
for  i  1  to  nua  do  begin 
for  j  :*  1  to  nua  do 
irite(outaatrix,c[i, j]:6:2); 

«riteln(outaatrix,RaMVotesCi]:6:2,M[il:8:3) ; 

end;  It  for  i  t) 

iriteln(outaatrix,’2eta(u)  =  ’,ZetaU[cluster,filenuel:6:4,’  Zetalc)  = 
ZetaC[cluster,filenual:6:4); 

uritelnloutaatrix,’ - ':30); 

end;  (t  procedure  OispiayVotes  t) 

procedure  Cal cZeta ( RonV: col uen; nua: i nteger ) ; 

It  ftlso  need  ’First’  froa  FindHeights  to  Find  a  corrected 
Zeta  for  tied  cases.  Also  verifies  totals  are  appropriate 
to  the  Kendall  assuaptions  t) 
var 

halfNav,i,j,NuaTies  ;  integer; 

TieLocate  ;  array  tO..Ul  of  integer; 
teap  :  col uen; 

verify, bogus, LoaSuaO,  HighSuaD,  AveSuaD  :  real; 

(mttttttmmittitt) 

procedure  FindLoaSuaD; 
var 

LoSuaD  :  real; 
begin 

for  i  :=  1  to  nua  do 
teaplil  :=  orderedlil; 
for  1  :=  1  to  halfaay  do 

teaplTielocateCill  roundlteapCTieLocatelill  *  0.5); 

It  since  in  order  of  big  to  saall,  to  obtain  a  large  estiaate 
of  the  nueber  of  triads  need  to  round  up  larger  values 
ithe  first  half)  and  round  doun  the  saaller  values  I) 
for  i  :=  halfaay+1  to  NuaTies  do 
teaplTieLocateli 1)  ;=  round) teaplTieLocatelill  -  0.5); 
verify  :=  0; 
for  i  1  to  nua  do 
verify  :=  verify  +  teapCi]; 
bogus  :=  nuat(nua-l)/2; 

if  verify  bogus  then  witelnrSua  of  revised  roavotes  (rounded  high) 
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veri{y:7:l,’  Hhich  should  be  •  ’,boqus:7:l); 

LoSuaO  0;  (t  Dobbins  convention  since  high  estieete  gives 
loH  zeta  t) 
lor  i  :=  1  to  nue  do 
LoSueO  :=  LoSueO  *  teepCilKteeplil-l); 

LonSuiD  :=  first  -  LoSuaO/2| 
end;  (t  procedure  FindLouSueO  I) 

(tmttttttttttti) 
procedure  FindHighSuaO; 
var 

HiSuaD  :  real; 
begin 

for  i  1  to  nua  do 
teapCil  :=  orderedCil; 
for  i  1  to  halfnay  do 

teaplTieLocateCill  rounddeapCTteLocatelill  -O.S): 

for  1  halfMav>l  to  NuaTies  do 
teaplTieLocateCill  roundlteapCTieLocateCill  *0.5); 

verify  !=  0; 
for  1  1  to  nua  do 

verify  :*  verify  ♦  teaplil; 
bogus  i-  nuat(nua-l)/2; 

if  verify  <>  bogus  then  uritelnl'Sua  of  revised  rou  votes  (rounded 
Mou)  *  ’,yerify:7:l,’  uhich  should  be  =  ’,bogus:7:l); 

HiSuaO  :=  0; 
for  i  I  to  nue  do 
HiSuaO  NiSuaD  *  teapCi lt(teap(il-t); 

HighSuiD  first  -  HiSuaD/2; 
end;  (t  FindHighSuaD  t) 

(tntiitttttttttt) 
begin  It  procedure  CalcZeta  t) 
for  I  :=  1  to  nua  do 
orderedCil  :=  RonVCil; 

sort (ordered, nua);  (I  causes  ’i’  index  to  be  fro#  largest  to  saallest  I) 

NuaTies  0; 

for  1  :=  1  to  nua  do 

if  trunc (orderedCil)  <)  orderedCil  then  begin 
(I  1/2  reflects  tied  ranks  t) 

NuaTies  :=  NuaTies  »  1; 

TieLocateCNuaTiesl  i; 

It  TieLocate[l..NunTiesl  =  rou  I  of  tied  score  t) 
end;  (t  if  t  incidentally,  for  i  II 
halfuav  ;•  round (NuaTles/2); 

FindLonSuaD; 

FindHighSuaD; 

AveSuaD  :=  iHighSuaD  >  LcHSuaD)/2; 

AveZeta  1  -  AveSuaO/aaxtriadsinua); 
end;  (>  orocedure  CalcZeta  I) 

procedure  OoOneFi  leundeii :  integer  I ; 
begin 
crit  0; 
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reset  (Ntri  X 1 ,  i  nf  i  1  e[  index  ]> ; 

readln(e4trixl,l«t);  (I  last  =  criterion  ai  last  aatrix  I) 

readlnlaatrixl, author); 

readlnlaatrixl.dieentll); 

if  HantCopy  then  begin 

Nritelnl ’Judge  t’:20,index:3,’  Results  (Voting  Matrix)’); 
iritelnlouteatrix,’Judge  •’,index;3,’  Results  (Voting  Matrix)’); 
end;  (t  if  t) 
cluster  1; 

criterion!?]  ;=  ’bogus’;  «  inits  'last'  criterion  to  coaplete  repeat  t) 
repeat 

Nhile  die’^nlcluster]  «  0  do  readln(satrixl,diaen[cluster]); 

(I  the  ’0’  are  for  sequencing  coeposite  ueights  -  soeething 
this  progae  is  not  interested  in  doing  I) 

6etMatr IX (dieenlclusterl) ; 

FindWeiqht5(dieen[clu5ter]);(t  out:  uCilrpercentage  votes  ti  Rouvotes  ») 
CalcZeta(RQ«Votes,dinen[cluster]);  (I  outputs  average  zeta  t) 
ZetaCCcluster,filenui]  :=  AveZeta; 
i<  dantCopv  then  DisplayVotesidieenCclusterl); 

;f  criterionlclusterlOlast  then  begin 
cluster  :=  cluster  ♦  1; 
readln(eatrixl,dieen[cluster])i 
end:  it  if  t) 

until  criterionCclusterl  =  last; 
clo5e(eatrixl); 

and;  'I  procedure  DoOneFile  I) 

procedure  DisplaySueearylJudgeNua: integer); 

(t  urite  out  individual  votes,  total  priorities,  and  variance 
between  priorities  for  judges 
i) 
var 

filen, start, cease, crit,i,j  :  integer; 

SuaSqr,rotZeta,SuaO  :  real; 
total, MeanVote  ;  coluan; 
tuwtnutuutu) 
procedure  FmdClusterHeights; 
var 

laapos  ;  real; 
begin 
1  :=  1: 

for  crit  :=  start  to  cease  do  begin 
V'oteCountCcrit,14]  :=  Oj  (»  init  total  to  zero  I) 

SunSqr  :=  0; 

RowV'otesCil  :=  0; 

SuaD  :•  0;  (t  sase  as  local  variable  in  FindNeights  t) 
nuapos  :=  dieenlclusterl  t  (dieenlclusterl-U/Z; 

(t  nuapos  IS  nuaber  of  possible  votes  tor  single  voter  t) 
for  filer  :=  I  to  JudgeNua  do  begin 
VoteCountCcrit.lfJ  :=  VoteCountCcrit.MJ  +  VoteCountCcrit.filen); 
SuaSqr  :=  SuaSqr  *■  sqr(VoteCountCcrit,filenl/nuapos/; 
end:  (I  for  filen  ») 
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fteanVoteCi]  VoteCount[crit,14]/liait; 

(t  Insures  HeanVote  in  coluen  lorut  lor  CalcZeta  t) 
VoteCountIcritilS]  sqrt(SueSqr/liait  -  sqr(lteanVote[i]/nuapos))tlO; 
SuiO  :=  SueD  *  neanVateCilKNeanVotelil*!); 
i  :=  i+l; 

total [cluster]  :=  totallcluster]  >  VoteCount[crit,14]; 
end;  (t  for  crit  <) 

first  :=  dieen[cluster]t(diaenCcluster}-l)t(dieen[cluster]-2)/&; 

CaeZetaU  :=  (first-SuiO/2)/aaxtriads(diien[cluster]); 
end;  (t  procedure  FindClusterNeights  t) 

{tmutmmumutu) 
procedure  DisplayCluster; 

(I  VoteCountl ’criterion",  1..13]  =  judges, 14=total,15=variance  t) 
var 

objective  :  integer; 
begin 

t*rite(outeatrix,’Crit’,’r:4); 

for  i  2  to  JudgeNua  do  Hrite(oututrix,i:S); 

iriteln(outeatrix,’Totitt’:7,’DevtlO’:7); 

objective  :=  0;  it  to  break  crit  into  cluster  criterion  nuabers  ») 
for  crit  :•  start  to  cease  do  begin 
objective  :=  objective  +  1; 
writetouteatr IX, objective: 3,’  ’) ; 
for  filen  1  to  JudgeNua  do 
ariteioutiatrix.VateCountCcrit,filen]:5:l); 
arit9(out#atrix,VoteCount[crit,l4]/totaHcluster]:5:3); 
ariteln(outiatru.VoteCount[crit,I5]:5:3);(l  DevIlO  between  judges  1) 
end;  (I  for  crit  t) 

TotZeta  0; 
i*rite(outeatrix,’Z(u)’); 
for  filen  :=  1  to  JudgeNue  do  begin 
*<rite(outeatrix,ZetaU[cluster,  filen]:  4:2) ; 

TotZeta  :=  TotZeta  ♦  ZetaUIcluster, filen]; 
end;  (t  for  t) 

«iriteln(out«atrix,TotZc'fa/Liait:5:3); 

TotZeta  :=  0; 

"nte(outaatrix,'Z(c)’); 

for  *!len  :=  1  to  JudgeNua  do  begin 

Hrit9(out«atrix.2etaC[cluster,filen]:4:2); 

TotZeta  :=  TotZeta  *  ZetaCCcluster, filen]; 
end;  (I  for  I) 

writeln(outoatrixJatZeta/JudgeNui:5:3l: 
end;  II  procedure  DisplayCluster  I) 
ilttllltttltllttltti; 
begin  (t  DisolavSuaoarY  I) 
renriteloutaatrix, 'IjiSueVotes.  text’) : 

ariteln (outoatrix, 'Voting  Suaaarv’iZO, '  for’ ,litit:3.’  Decision  Makers’); 
start  :=  1;  It  Begins  neu  cluster  in  ’for"  loop  *i 
♦or  cluster  I  to  *?  do  begin 
total  [cluster]  ):  ii  votes  per  cluster  t.‘ 
cease  start  +  fiiaentciuster]  -  1: 
ariteln  lout  flat  r  IX,  cri  teriorilclusterZi; 
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Hriteln'v'iterion(cluster]); 

FindClusterHeifhts;  (t  returns  NeanVoteCi]'  and  ’First’  t) 
CalcZetalHeanVote, diaentcluster ]) ; 

DisplayCluster; 

start  :=  cease  *  1;  <t  avoids  printing  sate  line  teice  I) 
end;  (t  For  cluster  t) 
closeloutMtrix,lock); 
end;  (t  procedure  DisplaySuaaary  t) 

procedure  InitOld; 
begin 

inFilel  11  ’I5:arnettin.text’;  inFilet  21  :=  ’#S:aochelin.text’; 

inFileC  31  :=  ’I5:parrinqin.text’;  inFilet  41  ;=  'iSrperenicin.texf; 

inFileC  51  :=  ’tS:rallin.text’:  inFilet  61  :=  ’iStrobackin.text’; 

inFilet  71  :=  ’I5:ricein.text’;  inFilet  81  :=  ’Mistanleyin.text’; 

inFilet  ?!  :=  ’ISiMalbergin.text’ ;  inFiletlOl  :=  ’t5:broMnin.text’; 
inFiletlll  :=  ’ISihartin.text’;  inFiletl2]  :=  ’I5:hau5erin.text’; 

inFiletl3]  :=  ’ISigeoaean.text’; 
e,.:;  (I  InitOld  I) 

procedure  InitNee; 
begin 

inFilet  11  i=  ’•Siarnettinl.text’ ;  inFilet  21  :*  ’ISiaochelinl.text’; 

inFilet  31  :=  ’I5:parrinqinl.text’ i  inFilet  41  :=  ’I5:perenicint.text’; 

millet  51  ’#5:rallinl.text’i  inFiIet  61  :=  ’#5:robackinl.text’; 

inFilet  71  t-  ’ISiriceinl.text’;  inFilet  81  ’I5:stanleyin».text’i 


procedure  InitOld; 

(t  sets  uD  Files,  note:  apple  speciFic  naaes.  Need  to  lable  Files 
ever  systee  calls  then. I) 
begin 

inFilet  11  ::  ’IS: judgel.text' ;  inFilet  21  ;=  ’45: judgeZ.text’; 

inFilet  31  :=  ’15: judge3.text’;  inFilet  41  :=  ’15: judgel.text’; 

inFilet  51  :=  ’15: judgeS.text’ ;  inFilel  61  :=  ’#5: judges. text’ ; 

inFilet  71  :=  ’15: judge?. text’;  inFilet  81  :=  ’I5:judge8.text’; 

inFilet  91  ;=  ’15: judge?. text’ ;  inFiletlOl  :-  ’15: judgelO.text’; 
inFiletlll  ’15: judgell.text’;  inFilell21  :=  ’I5;judgell.text’; 
inFiletl31  ;=  ’t5:geoaean.test’; 
end;  t  InitCld  tl 


procedure  InitNeu; 
begin 

.r.filel  11  :=  '15:  judgell.text’; 
inFilet  31  :=  ’K: judge3t.text’; 
inFilet  51  ’15: judge5l.text’ ; 

inFilet  71  :=  ’I5:judge7l.text’: 
inFilet  91  i-  '15: judge9*.text’ ; 
inFiletlll  :=  ’t5:judgellt.text’; 
irFiletl31  't5:geoaeant.text’ ; 
end;  t  InitNew  II 


inFilet  21  :=  ’15: ,udge2l.text’ ; 
inFilet  41  :=  ’15: judge4t.tex t’ 
inFilet  61  :=  ’ISijudgebt.text’, 
inFilet  81  :=  ’15: jUdgeSl.text’ 
inFiletlOl  :=  ’15: judgelOl.text 
inFiletl2]  :=  ’I5:judgelll.text 
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procedure  SetUp; 

(I  Nakes  Headings  and  asks  user  for  his  inputs  (liaits,  uantcopy)  t) 
var 

ans.neutile  :  string; 
begin 

triteCDo  you  uant  neu  or  old  files?  In  or  o)  ’IsreadlnlneMfile); 
if  neufile  =  ’o’  then 
InitOld 
else  begin 
InitNeu; 

uritelnl’  Using  New  Files’); 
end:  (I  if  t) 

Naaeit;  (t  puts  criterion  titles  in  an  array  for  display  I) 

NTiteCHoH  eany  files  do  you  uant  to  look  at  (13=geoeean)?  ’); 

readln(liiit); 

if  lieit  =  1  then  begin 

Nritel’Hhich  file  nuaber  do  you  uant  to  work?  ’);  readln(filenui); 
infiletll  :=  infilelfilenuel;  (I  if  1  file,  only  infiietlJ  read  I) 
end;  (t  if  I) 

uritelnCFile  VotesOut  Mill  contain  :eta  and  average  weights  and  votes’); 
writel’Do  you  want  individual  aatrices  -  file  zetaoutly  or  n)?  ’I; 
read! n (ansi; 

if  'ans  =  ’v’)  or  (ans  =  ’Y’)  or  (ans  *  ’yes’)  then 
bantCopy  ;=  true 
else 

HantCopy  false; 
end;  (I  SetUo  I) 

begin  (t  aain  prograa  Zeta  tl 
rewritelouteatrix, ’fSizetaout.text’ ) ; 

Setup;  (I  initializes  files,  user  inputs  lint  &  NantCopy  t) 
writelnl’Standby  this  takes  awhile’); 
for  filenua  :=  1  to  halt  do 
OoOneFiletfilenui) ; 
close(outaatnx,lock); 

SisplaySueearydieit);  (t  writes  to  file  VotesOut  t) 
end.  (t  earn  prograa  Zeta  I) 
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Sanole  RaN  Inout  File 


.laneuverabilitv  -  *  -  Designates  the  Last  Criterion  llatrix  to  be  read  in  (for  shorter  runs). 

Judge  I  31  -  Designates  the  'Title"  or  Judge’s  Naee  H=  Revised  Data). 

a  -  -  -  Designates  the  Dieension  of  HatriK  about  to  be  read. 


113  5  7 
13  5:’ 

3  7  7 
5  7 


-  AHP  CoeoariBons  for  Half  Matrix 


TftV  Effectiveness  -  -  Designates  Criterion  of  Matrix  Just  Head. 
3 


7  3  5  5  9  -3  1 
3-3-5  5  -5  -5 
-3  -7  3  -5  -3 


_e  c  .5 


-Q  -0 


Reouireaent 5 

e  ^ 

1 

Perforeance 

0  -  -  Zeros  for  Seouencing  Ccioosite  Heights 


L=f5 


-OSiS 
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APPENDIX  D:  Sample  Computer  Input-Output 


j 

■) 

'"1 

0 

0 

0 

0 

0 

0 

0 

A 

V 

0 

0 


®r:ceU3nt5 

0 


P!aintaiia3ilit. 

0 

C 


'^ansu»9''aDi  1 1 1  < 


D-2 


1 

RD-A155  858  RGGREQRTING  INDIVIDUAL  PREFERENCES  IN  THE 
HIERRRCHV  PROCESS  APPL.  .  <U>  AIR  FORCE  INST 
HRIGHT-PATTERSON  AFB  OH  SCHOOL  OF  ENGl.  . 
UNCLASSIFIED  15  HAR  85  AFIT/GST/0S/85H-12 

RNALVTIC 

OF  TECH 
>  E  REED 

F/G  9/2 

^4  'T| 

.4 

■ 

ii 

1 _  - _ 1^ 

MICROCOPY  RESOLUTION  TEST  CHART 

NAIIUNAL  BURtAU  OF  SIANOARDS-1961  A 


APPENDIX  D:  Sample  Computer  Input-Output 


Risks,  Factor  I  4  Level  #  3 


I 

3 

Weight  Coao-Wt 

1.00 

0.20 

0.14 

0.07193  0.0095 

5.00 

1.00 

0.33 

0.27895  0.0370 

7.00 

3.00 

1.00 

0.64912  0.0860 

laiida(iax):  3.01)49 

C.I.:  0.03244  C.R.: 

LoQl sties. 

Factor  t  5  Level  R  3 

1 

n 

3 

Height  Coeo-Ht 

1.00 

5.00 

0.33 

0.27895  0.0134 

0.20 

1.00 

0.14 

0.07193  0.0034 

3.00 

7.00 

1.00 

0.64912  0.0311 

laadalaax):  3. 

0649 

C.I.!  0.03244  C.R.: 

Survivahilitv, 

Factor  1  6  Level  * 

7 

j 

1 

n 

J 

4 

5 

6 

7  Height 

CoiD-Wt 

1.00 

7.00 

3.00 

7.00 

0.33 

5.00 

0.20  0.17865 

0.0050 

0.14 

1.00 

0.33 

3.00 

0.20 

1.00 

0.20  0.04804 

0.0013 

0.33 

3.00 

l.OO 

3.00 

1.00 

3.00 

0.20  0.10180 

0.0029 

0.14 

0.33 

0.33 

1.00 

0.20 

3.00 

0.14  0.03947 

0.0011 

3.00 

5.00 

1.00 

5.00 

1.00 

5.00 

0.13  0.19953 

0.0056 

0.20 

1.00 

0.33 

0.33 

0.20 

1.00 

0.14  0.03310 

0.0009 

5.00 

5.(00 

5.  (NO 

7.00 

3.00 

7.00 

1.00  0.39935 

0.0112 

landafia/:):  7 

.3842 

C.I.: 

0. 14736 

C.R. 

:  0.11164 

C  5.  1. 

should 

be:  1. 

12  actual:  1  5, 

11  =  1 

.0  Kith  C.R.s  0.08807 

Survivability. 

Factor 

t  b  Level  k 

y 

i 

1 

2 

•y 

w 

4 

5 

6 

7  Weight 

Coio-Wt 

1.00 

7.00 

3.00 

7.00 

1.00 

5.00 

0.20  0.2016^ 

0.0057 

0.14 

1.00 

0.33 

3.00 

0.20 

1.00 

0,20  0.04976 

0.0014 

0. 33 

!.00 

1.00 

3.00 

1.00 

3.00 

0,20  0.10187 

0.0029 

0.14 

0.33 

V.  33 

1.00 

0.20 

3.00 

0,14  0.04068 

0.0011 

1.00 

5. 00 

1 . 00 

5.00 

1.00 

5.00 

0.33  0.15885 

C.0045 

0.20 

1.00 

0.33 

0.33 

0.20 

1.00 

0,14  0.03410 

0.0010 

5.00 

5.00 

5.00 

7.00 

3.00 

7.00 

1.00  0.41307 

0.0116 

landaieax):  7 

.6975 

C.I.: 

0.11626 

C.R. 

:  O.C8807 
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APPENDIX  D:  Sample  Computer  Input -Output 


Propellants  ,  Factor  #17  Level  i  ♦ 


1 

2  3 

Height 

1.00 

l.OO  0.33 

0.20000 

1.00 

1.00  0.33 

0.20000 

3.00 

3.00  1.00 

0.60000 

laadaleax):  3.0000 

C.I.:  0.00000  C.R.; 

0.00000 

llaintainabilitY. 

Factor  #19  Level  #  4 

1 

2  3 

4  Height 

1.00 

0.33  0.20 

1.00  0.10685 

3.00 

1.00  0.33 

1.00  0.20979 

5.00 

3.00  1.00 

3.00  0.52550 

1.00 

1.00  0.33 

1.00  0.15786 

laadaleax):  4.1155 

C.I.:  0.03849  C.R.: 

0.04276 

.naneuverability. 

Factor  #  I  Level  #  5 

i 

•  V 

Height 

1.00 

5.00  7.00 

0.71471 

0.20 

1.00  5.00 

0.21849 

0.14 

0.20  1.00 

0.06680 

laida(«ax)!  3.1928  C.I.!  O.O’lia  C.R.:  0.15756 

i  3.  11  should  be;  0.09  actual;  1  1.  31  *  ?.0  with  C.R.»  0.10095 
[  3.  11  s-Oould  be;  0.08  actual;  I  1.  21  *  4.0  mth  C.R.=  0.06144 

tlaneuverabihty.  Factor  I  1  Level  #  5 


1  2  3  Meiqht 

1.00  4.00  9.00  0.70852 

7.25  l.CO  5.00  0.23115 

O.n  0.20  1.00  0.06033 

landalia';.!  3.0713  C.!.:  O.035o3  C.R.; 

^ne  Consistency  of  tnis  nierarchv  =  0.0660 


O.Od144 
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APPENDIX  E:  Statistical  Tests 


Significance  of  tau,  concordance  (W> ,  and  zeta 
HosMeasure  is  not  significant.  If  P  >  a,  accept  Ho,  else  reject 


APPENDIX  E:1 


Probabilities  Associated  with  Values  as  Lau^ge  as  Observed 
Values  of  S  in  the  the  Kendall  Rank  Correlation  Coefficient 


s 

Values  of 

N 

S 

Val ues 

of  N 

4 

5 

8 

6 

7 

(6}« 

(10)« 

(28)* 

(IS)* 

(21)* 

0 

.625 

.592 

.548 

1 

.500 

.500 

2 

.375 

.408 

.452 

3 

.360 

.386 

4 

-  167 

.242 

.360 

5 

.235 

.281 

6 

.042 

.  117 

.  138 

7 

.136 

.  191 

8 

.042 

.199 

9 

.068 

.119 

10 

.008 

.138 

11 

.028 

.068 

12 

.089 

13 

.008 

.035 

14 

.054 

15 

.001 

.015 

16 

.031 

17 

.054 

18 

.016 

19 

.005 

20 

.007 

21 

.001 

«<a) 

-  conversion 

1  factor: 

S  =  (a) 

(tau) 

(Siegel,  1956:285) 
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APPENDIX  E>2 


Critical  ValUM  of  S  in  th«  Kandall  Coafficiant  of  Concordanca 

□bjacti vaB(k) 


N 

3 

4 

5 

6 

7 

( 

.05  aignif icance) 

3 

64.4 

103.9 

157.3 

4 

49-5 

88.4 

143.3 

217.0 

5 

62.6 

112.3 

182.4 

276.2 

6 

75.7 

136.1 

221.4 

335.2 

B 

48.1 

101.7 

183.7 

299.0 

453.  1 

( 

.01  Bignif icanca) 

3 

75.6 

122.8 

185.6 

4 

61.4 

109-3 

176.2 

265.0 

S 

80.5 

142.8 

229.4 

343.8 

6 

99.5 

176.1 

282.4 

422.6 

B 

66.8 

137.4 

242.7 

388.3 

579.9 

(Siegal , 

1956:  2f 
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APPEI^IX  E:3 


Sagni-f icancB  of  Zsta,  for  N  ■  3  through  9 


N  «  3 

N  « 

4 

N  B 

5 

N  O 

•  / 

Zata 

P  Zata 

P 

Zata 

P 

Zata  P 

Zata 

P 

0  1 

.0  0 

.625 

0 

1.0 

0  1.0 

0 

1.0 

1.0 

0  .50 

.375 

.20 

.703 

.125  .773 

.072 

.964 

1.0 

0 

.40 

.469 

.250  .509 

.143 

.853 

N  ^ 

'  8 

.60 

.234 

.375  .398 

.214 

.737 

Zata 

P 

.80 

.117 

.500  .208 

.286 

.553 

0 

1.0 

1.0 

0 

.625  .120 

.357 

.420 

.050 

.949 

.750  .051 

.429 

.287 

.  100 

.859 

.875  .022 

.500 

.  198 

.ISO 

.768 

1.0  0 

.572 

.  112 

.200 

.629 

.643 

.069 

.250 

.520 

.715 

.  033 

.300 

.390 

.787 

.017 

.350 

.299 

.858 

.006 

.400 

.208 

.929 

.002 

.450 

.  153 

.500 

.094 

.550 

.063 

.600 

.037 

.650 

.023 

.700 

.011 

.750 

.006 

.800 

.003 

(Dobbins^  l?80b:32) 
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APPENDIX  F:  AHP  Summary  of  Priorities 


AHP  SuHary  lor  13  Decision  HakKS 
Judge  113  =  Geoeetric  Mean  Input  lor  the  First  9  Judges 
Nean  »  Arithaetic  Mean  of  Individual  Priority  Vectors 
TAV  Effectiveness 

Crit  1  2  3  4  5  6  7  8  9  10  11  12  13 

1  0.313  0.159  0.259  0.600  0.166  0.181  0.611  0.241  0.040  0.345  0.395  0.211  0.278 

2  0.141  0.203  0.259  0.127  0.234  0.278  0.149  0.269  0.323  0.100  0.326  0.386  0.263 

3  0.167  0.142  0.274  0.040  0.268  0.044  0.026  0.212  0.074  0.084  0.022  0.188  0.123 

4  0.051  0.069  0.133  0.104  0.101  0.073  0.083  0.033  0.270  0.066  0.074  0.051  0.102 

5  0.244  0.151  0.048  0.047  0.114  0.031  0.037  0.184  0.190  0.166  0.132  0.12S  0.109 

6  0.083  0.275  0.028  0.083  0.117  0.393  0.094  0.061  0.103  0.239  0.051  0.039  0.125 

Cl  0.069  0.062  0.056  0.198  0.082  0.310  0.147  0.095  0.121  0.141  0.319  0.248  0.011 

CR  0.056  0.050  0.045  0.160  0.066  0.250  0.119  0.077  0.097  0.114  0.258  0.200  0.009 

Mean  Dev  t  10 
0.293  1.37 

0.233  0.52 

0.128  0.77 

0.092  0.51 

0.123  0.54 

0.131  0.97 

0.143 
0.115 


Requireeents 

Crit  1  2  3  4  5  6  7  8  9  10  11  12  13 

1  0.134  0.164  0.239  0.254  0.129  0.237  0.410  0.075  0.246  0.179  0.422  0.136  0.230 

2  0.134  0.187  0.044  0.040  0.215  0.075  0.121  0.208  0.132  0.108  0.146  0.018  0.134 

3  0.155  0.175  0.035  0.028  0.140  0.036  0.036  0.018  0.019  0.074  0.024  0.069  0.061 

4  0.155  0.183  0.062  0.332  0.058  0.044  0.174  0.129  0.150  0.234  0.242  0.063  0.145 

5  0.155  0.029  0.173  0.183  0.324  0.115  0.105  0.205  0.076  0.082  0.047  0.295  0.150 

6  0.027  0.029  0.016  0.067  0.036  0,014  0.027  0.015  0.030  0,051  0.013  0.086  0.032 

7  0.084  0.099  0.302  0.011  0.033  0.390  0.029  0.111  0.063  0.076  0.021  0.185  0.093 

8  0.155  0.135  0.128  0.086  0.064  0.089  0.097  0.238  0.234  0.197  0.085  0.M8  0,154 

Cl  0.034  0.036  0.172  0.466  0.090  0.317  0.087  0.157  0.140  0.104  0.441  0.840  0.021 

CR  0.024  0.026  0.122  0.330  0.064  0.225  0.062  0.111  0.099  0.074  0.313  0.596  0.015 

Mean  Dev  I  10 

0.219  0.78 

0.119  0.50 

0.067  0.49 

0.152  0.69 

0.149  0.75 

0.034  0.19 

0.117  1.04 

0.142  0.48 

0.224 
0.159 
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APPENDIX  F:  AHP  Summary  of  Priorities 

PerfortMce 

Crit  1  2  3  4  5  6  7  B  9  10  11  12  13 

1  0.429  0.429  0.659  0.191  0.443  0.564  0.600  0.685  0.158  0.097  0.753  0.487  0.489 

2  0.429  0.143  0.156  0.048  0.169  0.172  0.200  0.234  0.076  0.202  0.184  0.078  0.181 

3  ).143  0.429  0.185  0.761  0.387  0.264  0.200  0.080  0.766  0.701  0.063  0.435  0.330 

Cl  0.000  0.000  0.015  0.164  0.009  0.788  0.000  0.147  0.068  0.068  0.147  0.006  0.002 

CR  0.000  0.000  0.025  0.282  0.016  1.359  0.000  0.254  0.117  0.117  0.254  0.011  0.004 

Heui  Oev  I  10 

0.458  1.51 

0.174  0.76 

0.368  2.14 

0.109 
0.188 

Risks 

Crit  1  2  3  4  5  6  7  8  9  10  11  12  13 

1  0.091  0.088  0.072  0.722  0.550  0.778  0.333  0.132  0.308  0.200  0.067  0.078  0.301 

2  0.455  0.669  0.279  0.227  0.210  0.111  0.333  0.694  0.308  0.400  0.218  0.435  0.401 

3  0.455  0.243  0.649  0.051  0.240  O.lll  0.333  0.174  0.385  0.400  0.715  0.487  0.298 

Cl  0.000  0.004  0.032  0.104  0.009  0.000  0.000  0.040  0.000  0.000  0.091  0.006  0.000 

CR  0.000  0.006  0.056  0.180  0.016  0.000  0.000  0.069  0.000  0.000  0.158  0.011  0.000 

Hean  Oev  t  10 

0.285  2.27 

0.362  1.32 

0.354  1.60 

0.022 

0.038 

Loqi sties 

Crit  1  2  3  4  5  6  7  B  9  10  11  12  13 

1  0.143  0.258  0.279  0.048  0.163  0.203  0.455  0.134  0.600  0.130  0.633  0.055  0.231 

2  0.429  0.105  0.072  0.191  0.297  0.055  0.091  0.119  0.200  0.062  0.063  0.203  0.160 

3  0.429  0.637  0.649  0.761  0.540  0.742  0.455  0.747  0.200  0.808  0.304  0.742  0.609 

Cl  0.000  0.019  0.032  0.164  0.005  0.218  0.000  0.006  0.000  0.068  0.068  0.218  0.001 

CR  O.Ow  0.033  0.056  0.282  0.008  0.376  0.000  0.011  0.000  0.117  0.117  0.376  0.001 

Rean  Oev  t  10 

0.258  1.69 

C.157  0.95 

0.584  0.84 

0.061 
0.106 

Survivability 

Crit  1  2  3  4  5  6  7  8  9  10  11  12  13 

1  0.060  0.035  0.179  0.020  0.053  0.025  0.063  0.039  0.048  0.053  O.Olb  0.295  0.060 

2  0.144  0.078  0.048  0.019  0.295  0.095  0.128  0.364  0.093  0.041  0.449  0.261  0.126 

3  0.164  0.066  0.102  0.250  0.039  0.074  0.569  0.022  0.194  0.406  0.073  0.065  0.139 

4  0.144  0.092  0.039  0.055  0.107  0.046  0.045  0.238  0.194  0.191  0.149  0.093  0.108 

5  0.049  0.056  0.200  0.084  0.094  0.138  0.054  0.073  0.038  0.105  0.035  0.042  0.092 

6  0.219  0.214  0.033  0.280  0.051  0.294  0.063  O.ObO  0.252  0.121  0.036  0.043  0.159 

7  0.219  0.460  0.399  0.293  0.360  0.328  0.077  0.204  0.181  0.083  0.242  0.200  0.316 

Cl  0.072  0.109  0.147  0.208  0.  129  0.215  0.137  0.139  0.086  0.221  0.333  0.236  0.007 

CR  0.054  0.082  0.112  0.158  0.098  0.163  0.104  0.1-06  0.065  0.168  0.252  0.179  0.006 
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APPENDIX  F:  AHP  Summary  of  Priorities 

Hean  Dev  I  10 
0.074  0.72 

0.168  1.20 
0.169  1.46 

0.116  0.S2 

0.081  0.39 

0.139  0.88 

0.254  0.83 

0.157 
0.119 


Propellants 

Crit  1  2  3  4  5  6  7  8  9  10  11  12  13 

1  0.143  0.287  0.200  0.070  0.297  0.766  0.091  0.591  0.111  0.649  0.761  0.735  0.261 

2  0.429  0.078  0.200  0.672  0.163  0.076  0.455  0.261  0.111  0.072  0.048  0.058  0.259 

3  0.429  0.635  0.600  0.257  0.540  0.158  0.455  0.148  0.778  0.279  0.191  0.207  0.480 

Cl  0.000  0.047  0.000  0.218  0.005  0.068  0.000  1.115  0.000  0.032  0.164  0.059  0.026 

CR  0.000  0.081  0.000  0.376  0.008  0.117  0.000  1.923  0.000  0.056  0.282  0.101  0.046 

nean  Dev  I  10 

0.392  2.38 

0.219  1.73 

0.390  1.63 

0.133 
0.230 


llaintainabilitY 

Cnt  1  2  3  4  5  6  7  8  9  10  11  12  13 

1  0.094  0.091  0.107  0.059  0.136  0.075  0.063  0.047  0.045  0.645  0.034  0.119  0.083 

2  0.176  0.117  0.210  0.342  0.340  0.234  0.313  0.536  0.511  0.192  0.204  0.047  0.312 

3  0.483  0.396  0.526  0.121  0.287  0.613  0.313  0.298  0.292  0.044  0.683  0.298  0.369 

4  0.247  0.396  0.  158  0.478  0.237  0.078  0.313  0.119  0.153  0.119  0.080  0.536  0.236 

Cl  0.072  0.011  0.038  0.260  0.062  0.398  0.000  0.130  0.076  0.162  0.195  0.130  0.003 
CR  0.080  0.012  0.043  0.288  0.069  0.442  0.000  0.144  0.084  0.180  0.217  0.144  0.004 
Mean  Dev  t  10 

0.126  1.49 

0.269  1.13 

0.363  1.41 

0.243  1.27 

0.118 
0.131 

Maneuverability 

Cnt  1  2  3  4  5  6  7  8  9  10  11  12  13 

1  0.101  0.109  0.715  0.761  0.167  0.778  0.455  0.701  0.188  0.114  0.742  0.649  0.435 

2  0.674  0.163  0.218  0.191  0.667  0.111  0.455  U.240  0.731  0.481  0.203  0.072  0.410 

3  0.226  0.729  0.067  0.048  0.167  0.111  0.091  0.059  0.081  0,405  0.055  0.279  0.155 

Cl  0,043  0,043  0.091  0.164  0.000  0.000  0.000  0.147  0.032  0.015  0.218  0.032  O.ulO 

CR  0.074  0.074  0.158  0.282  0.000  0.000  0.000  0.254  0.056  0.025  0.376  0.056  0.017 

Mean  Dev  I  10 

0.457  2.41 

a.  350  1,96 

0.193  1.77 

0.061 
0.106 
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APPENDIX  G:  Voting  Summary  of  Priorities 


Voting  Suaury  for  13  Decision  Haters 
Judge  tl3  *  Converted  AHP  Geoaetric  Hean  Input 
TAV  Effectiveness 

Crit  1  2  3  4  5  6  7  8  9  10  11  12  13  Totdt  DevIlO 

1  4.5  2.5  4.0  5.0  2.5  3.0  5.0  3.5  0.0  5.0  5.0  3.5  5.0  0.249  0.948 

2  2.5  3.5  4.0  3.5  4.5  4.5  3.5  4.0  4.5  1.5  4.0  4.5  4.0  0.249  0.564 

3  3.0  2.0  4.0  0.5  4.5  1.0  0.0  3.5  1.0  1.0  0.0  3.5  1.0  0.128  1.011 

4  0.0  0.5  2.0  3.0  1.0  1.5  2.5  0.0  4.0  0.5  2.0  1.0  0.0  0.092  0.808 

5  4.0  2.5  1.0  1.0  1.0  0.5  1.0  3.0  3.5  3.0  3.0  2.5  3.0  0.149  0.747 

6  1.0  4.0  0.0  2.0  1.5  4.5  3.0  1.0  2.0  4.0  1.0  0.0  2.0  0.133  0.952 

2(u)  0.84  0.37  0.88  0.78  0.75  0.88  0.91  0.72  0.91  0.91  1.00  0.81  l.OO  0.827 

Z(c)  0.88  0.44  0.88  0.81  0.81  0.94  0.94  0.75  0.94  0.94  1.00  0.88  1.00  0.861 


Requireaents 

Crit 

1 

V 

4  5 

a 

7 

/ 

8 

9 

10 

11 

12 

13  TotHt  DevtlO 

1 

4.0 

4.5 

5.5 

b.O  4.5 

5,5 

7,0 

2.5 

6.0 

5.5 

7.0 

4.0 

7.0  0.190  0.467 

4.0 

5.0 

2.0 

2.0  6.0 

3.5 

4.5 

5.5 

4.5 

3.5 

5,0 

0.5 

3.0  0.135  0.541 

T 

4.5 

5.0 

1.0 

2.0  4.0 

1.5 

1.5 

0.5 

0.0 

2.0 

1.5 

1.5 

1.0  0.071  0.529 

4 

4.5 

5.0 

3.0 

6.5  3.0 

2.0 

5.0 

4.0 

5.0 

6.0 

6.0 

3.0 

6.0  0.162  0.493 

5 

4.5 

0.5 

5.0 

5.0  7.0 

4.0 

4.0 

6.0 

3.0 

2.5 

3.0 

7.0 

5.0  0.155  0.628 

6 

0.0 

0.5 

0.0 

3.0  1.0 

0.0 

1.0 

0,5 

1.0 

1.0 

0.5 

3.0 

0.0  0.032  0.351 

7 

2.0 

3.0 

7.0 

0.0  0.0 

7.0 

1.0 

3.0 

2.0 

2.0 

1.0 

5.0 

2.0  0.096  0.797 

3 

4.5 

4.5 

4.5 

7  C  C 

Wt  W  1  V 

4.5 

4.0 

6.0 

6.5 

5.5 

4.0 

4.0 

4.0  0.159  0.360 

Zlu) 

0.43  0.63  0.96  0.81  0.94  0.85  0.76  0.35  0.94  0.60  0.96  0.66  1.00  0.799 

Z'c) 

0.45  0.65  0.98  0.33  0.95  0.88  0.78  0.88  0.95  0.63  0.98  0.68  1.00  0.815 

Pertornance 

Cat 

1 

1 

A 

V 

4  5 

6 

7 

8 

9 

10 

11 

12 

13  Totdt  OevIlO 

1 

1.5 

1.5 

2.0 

1.0  1.5 

1.5 

2.0 

2.0 

1.0 

0.0 

2.0 

1.5 

2.0  0.500  1.849 

•s 

1.5 

0.0 

0.5 

0.0  0.0 

0.5 

0.5 

1.0 

0.0 

1.0 

1.0 

0,0 

0.0  0.154  1.662 

3 

0.0 

1.5 

0.5 

2.0  1.5 

1.0 

0.5 

0.0 

2.0 

2.0 

0.0 

1.5 

1.0  0.346  2.486 

:(u) 

0.  Z5 

0.75  0.75 

1.00  0.75  0.25  0.75  1.00 

1.00  1.00  1.00  0.75 

1.00  0.827 

;ic) 

1.00 

1.00 

1.00 

1.00  1.00  0.50  1.00 

1.00  1.00 

1.00  1.00  1.00 

1.00  0.962 

Risks 

:r;t 

1 

•n 

4  5 

6 

7 

8 

9 

10 

11 

12 

13  Totdt  DevtlO 

a 

1 

0.0 

0.0 

0.0 

2.0  2.0 

2.0 

1.0 

0.5 

0.5 

•).0 

0.0 

0.0 

0.0  0.205  2.708 

1.5 

:.o 

1.0 

1.0  0.5 

0.5 

1.0 

2.0 

0.5 

1.5 

1.0 

1.5 

2.0  0.410  1.804 

7 

1.5 

1.0 

2.0 

0.0  0.5 

0.5 

1.0 

0.5 

2.0 

1.5 

2.0 

1.5 

1.0  0.385  2.107 

Ziu.' 

0.75 

1.00 

1.00 

1.00  0.75 

0.75 

0.00  0.75  0.75 

0.75 

1.00  0.75 

1.00  0.788 

Zic) 

1.00 

1.00 

1.00  1.00  l.iX) 

1.00  0.00 

l.'OO 

I.UO 

1.00  1.00 

1.00 

1,00  0.923 

Logistics 

:r;t 

1 

■J 

4  5 

0 

8 

Q 

10 

11 

12 

13  Totrit  DevIlO 

1 

0.0 

1.0 

1.0 

0.0  0.0 

1.0 

l.S 

0.5 

2.0 

1.0 

2.0 

0.0 

1.0  0.282  2.301 

; 

1 . 5 

0.0 

0.0 

1.0  1.0 

0.0 

0.0 

0.5 

0.5 

0.0 

0.0 

1.0 

0.0  0.141  1.710 

3 

1.5 

2.0 

2.0 

2.0  2.0 

2.0 

1.5 

2.0 

0.5 

2,0 

1.0 

2.0 

2.0  0.57’  1.549 

;(u) 

0.75 

1.00 

1.00 

1.00  1.00 

1.00  0.75 

0.75  0.75  1.00 

1.00 

1.00 

1.00  0.923 

Z(c> 

1.00 

1 . 00 

l.'OO 

1,00  l.'iO 

1.00 

1.00 

1.0*7 

1.00 

1.00 

1.00 

1.00 

l.iXi  l.OC'O 
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APPENDIX  G:  Voting  Summary  o-f  Priorities 


Survivability 


Crit 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13  TotMt  OevtlO 

1 

1.0 

0.5 

4.0 

0.5 

2.0 

0.5 

2.5 

1.5 

1.0 

1.0 

0.0 

5.5 

0.0  0.073  0.744 

3.5 

3.0 

1.5 

0.5 

5.0 

3.0 

3.5 

5.5 

2.0 

0.5 

6.0 

5.0 

3.0  0.154  0.823 

3 

4.0 

2.5 

3.5 

5.0 

0.0 

2.5 

6.0 

0.0 

4.5 

6.0 

3.0 

2.5 

5.0  0.163  0.895 

4 

3.5 

2.5 

1.0 

2.0 

4.0 

1.5 

1.5 

5.0 

4.5 

4.5 

4.0 

3.0 

3.0  0.147  0.597 

5 

0.0 

1.5 

4.5 

3.0 

3.0 

3.5 

2.0 

2.5 

0.0 

3.5 

1.5 

0.5 

1.0  0.097  0.657 

6 

4.5 

5.0 

0.5 

5.0 

1.5 

5.0 

2.5 

2.0 

5.0 

3.5 

1.5 

0.5 

3.0  0.145  0.800 

7 

4.5 

6.0 

6.0 

5.0 

5.5 

5.0 

3.0 

4.5 

4.0 

2.0 

5.0 

4.0 

6.0  0.222  0.548 

Z(ii) 

0.68  0.79  0.89  0.91  0.84  0.61 

0.46  0.89  0.84  0.82 

0.98 

0.86  0.93  0.808  1 

Z(c)  0.71  0.82  0.93  0.93  0.86  0.64  0.50  0.93  0.86  0.86  1.00  0.89  0.93  0.835 


Propellants 

Crit 

1 

n 

j 

4 

5 

& 

7 

8 

9 

10 

11 

12 

13  TotUt  DevtlO 

1 

0.0 

1.0 

0.5 

0.0 

1.0 

2.0 

0.0 

1.5 

0.5 

2.0 

2.0 

2.0 

1.0  0.346 

2.571 

1.5 

0.0 

0.5 

2.0 

0.0 

0.0 

1.5 

1.0 

0,5 

0.0 

0.0 

0.0 

0.0  0.179 

2.308 

3 

1.5 

2.0 

2.0 

1.0 

2.0 

1.0 

1.5 

0.5 

2.0 

1.0 

1.0 

1.0 

2.0  0.474 

1.710 

;(u) 

0.75 

1.00  0.75 

1.00 

1.00 

1.00  0.75 

0.25  0.75 

1,00  1,00 

1.00 

1.00  0.865 

2(0 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00  0.50 

1.00 

1.00  1.00 

1.00 

1.00  0.962 

Maintainabihtv 

Crit 

1 

n 

y 

■j 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13  TotHt  DevtlO 

1 

0.0 

0.5 

0.5 

0.0 

0.0 

0.5 

0.0 

0.0 

0.0 

3a0 

0.0 

1.0 

0.0  0.071 

1.342 

l.o 

0.5 

1.5 

2.0 

a  e 

44  J 

2.0 

2.0 

3.0 

3.0 

2.0 

2.0 

0.0 

2.0  0.301 

1.407 

3.0 

2.  S 

3.0 

1.0 

2.0 

3.0 

2.0 

2.0 

2.0 

0.0 

3.0 

2.0 

3.0  0.365  1.445 

4 

:.o 

2.5 

1.0 

3.0 

1.5 

0.5 

2.0 

1.0 

1,0 

1.0 

1.0 

3.0 

1.0  0.263 

1.342 

Z(u) 

1.00  0.75  0.63 

1.00  0.63 

0.88  0.50 

1.00  1.00 

1.00  1.00 

1.00 

1.00  0.875 

2(0 

1.00  1.00  0.75 

1.00  0.75 

1.00  0.50 

1.00 

1.00 

1.00  1.00 

1.00  1.00  0.923 

flaneuverabilitv 

'rit 

1 

y 

■j 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13  TotWt 

DevtlO 

1 

0.0 

0.0 

2.0 

2.0 

0.5 

2.0 

1.5 

2.0 

1.0 

0.0 

2.0 

2.0 

2.0  0.436 

2.815 

n 

2.0 

1.0 

1.0 

1.0 

2.0 

0.5 

1.5 

1.0 

2.0 

1.5 

1.0 

0.0 

1,0  0,397 

1.910 

T 

1.0 

2.0 

0.0 

0.0 

0.5 

0.5 

0.0 

0.0 

0.0 

1.5 

0.0 

1.0 

0.0  0.167 

2.168 

Z(u) 

1.00 

1.00 

1.00 

1.00  0.75 

0.75  0.75  1.00 

1.00  0.75 

1.00  1.00  1.00  0.923 

Z(c) 

1.00 

1.00 

1.00 

1.00 

l.iXl 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00  1.000 
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APPENDIX  H:  Anova  Results 


rtNrtLV  'JIij 
CONSIST 
BY  JUDGE 
SIZE 
Cir4DEX 

»,♦»**»»»»»«*»»♦» 


lUUKCE  Ot-  '.'r-iK  I  rtT  1  On 

MAIN  Ei-t  ECTS 
JUUuL. 

c  I :  c 

CIAJCX 

1 !  J  T  T  I  QNS 
luCCf  SIZE 

JUCGL  CINDCX 

iI.:E  CINDEx 

:-W«Y  INfERACTIONS 

JuDL’t  jIZE  CINDEX 


EXI-'i-r-.iWEC 


XL  SI  D'J#-*L 


^JI  A_ 


Of-  variance 


$  t  *  *  * 

«  «  « 

y  »  »  «  « 

»  «  »  * 

«  »  « 

Sum  of 

SQUi-iRES 

DF 

MEAN 

SuUARE 

F 

SIGNIF 
OF  F 

.  9  35 

16 

.  058 

1 . 839 

j  ‘337 

.  £06 

1  1 

.  046 

1 . 448 

.  165 

.  340 

4 

.  005 

2.6  77 

.  036 

.u89 

1 

.  089 

2.790 

.  093 

2.112 

5  S' 

.  036 

1.127 

.  299 

.  Sfo6 

44 

.  021 

.  648 

.  945 

1 .  1 59 

1 1 

.  105 

3,318 

.  OOi  1 

.  u46 

4 

.012 

.  362 

.835 

,024 

44 

.  ij  1 9 

.509 

.974 

.024 

44 

.  019 

.  589 

.  974 

■ .  070 

119 

.  ‘033 

1 . 024 

.455 

; . OSO  va  .032 

;;..920  2:S  .032 


Where:  Consist 
Ci ndex 
Size 
Judge 


Zeta  and  Cl  Measures  of  Consistency 

Cl  =  1,  Zeta  =  2 

Matrix  Dimension  (3, 4, 6. 7, 8) 

Judge  Identifier  (1-12) 
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APPENDIX  I:  Initial  Survey  Data 


Survivability 

1  2  3  4  5  i  7  Priority 

l.OiX)  0.143  0,143  1.000  0.143  0.143  0.143  0.02112 

7.000  1,000  1.000  5.000  5,000  0.143  0.143  0,126«8 

7.000  1.000  1,000  1.000  5.000  0.143  0.143  0.10922 

1.000  0.200  1.000  1.000  1.000  0.143  0.143  0.C4127 

7.000  0.200  0.200  1.000  1.000  1.000  1.000  0.10087 

7.000  7.000  7.000  7.000  1.000  1.000  1.000  0.30027 

7.000  7.000  7.000  7.000  I.OOO  I.OOO  1.000  0.3002’ 

lanlaliaxi:  9.2968  C.I.:  0.38280  C.R.:  0.29000 


Prooel lants 

1  2  3  Priority 

1.000  7.000  7.000  0.76623 
’;).143  1.000  0.333  0.0’590 
0.143  3.000  1,000  0.15787 
lam)a(»a,<):  3,  1356  C.l.i  0.06731  C.ft.:  0.11691 


flaintainaiilitv 

1234  Priority 
1.000  0.143  0.333  1.000  0.08215 

’.000  1.000  0.200  5.000  0.2P'S'.’ 

3.000  5,000  1.000  3.000  0.53430 

1.000  0.  200  0.333  1.000  0.08575 

lamdaUax)!  4.’2?6  C.I.:  0.24319  C.«.:  0.2'022 


lanetiver  ability 
1  2  3  Priorit. 

l.)00  ’.000  7.000  0.777’3 
0.143  1.000  1.000  O.lllll 
0.143  1.000  l.OO'O  0.11111 

lantdaiiay;:  3.0'000  C.I.:  C.OOC'OO  3.P.:  O.uOO'OO 

’“0  Ocnaisterc,  tt  this  haerarthy  =  0.4333 
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APPENDIX  Ij  Initial  Survey  Data 


Judge  i  6  AHP  RESULTS  (ORIGINAL  SURVEY) 
TAV  Effectiveness 


1 

'} 

3 

4 

5 

6 

Priority 

1.000 

0.333 

7.000 

7.000 

7.000 

0.143 

0.18195 

3.000 

1.000 

7.000 

5.000 

7.000 

1.000 

0.27484 

0.143 

0.143 

1.000 

7.000 

0.333 

0.143 

0.06890' 

0.143 

0.200 

0.143 

1.000 

5.000 

0.143 

0.05204 

0.143 

0.143 

3.000 

0.200 

1.000 

0.143 

0.04643 

7.000 

1.000 

7.000 

7.000 

7.000 

1 . 000 

0.37583 

afflda(iax):  8 

.2378 

C.I.: 

0.44755  C.R, 

.:  0.36093 

1 

Reauireeents 

4  5 

s 

7 

8 

Priority 

1.000 

^000 

7.000 

7.000  7.000 

9.000 

0.200 

1.000 

0.23927 

0.143 

1.000 

5.000 

5.000  0.200 

7.000 

0.143 

1.000 

0.08153 

0. 143 

0.200 

l.OOiO 

0.200  I.OOO 

5. 000 

0. 143 

0. 333 

0.03759 

0.143 

0.200 

5.000 

1.000  0.200 

5.000 

0.143 

0.333 

0.04766 

0.143 

5.000 

1.000 

5.000  I.OOO 

5.000 

0.200 

1.000 

0. 10452 

0.111 

0. 143 

0.200 

0.200  0.200 

1 . 000 

0.111 

1.000 

0.02127 

5.000 

7.000 

’.000 

7.000  5.000 

9.000 

l.OOO 

C.O'OO 

0.38756 

1 . 000 

1.000 

3.000 

3.000  1.000 

1.000 

0.143 

I.OOO 

0.06062 

aiida'iax':  10 

.3973 

L .  I .  I  v' .  j4i4 

1’  C.I 

R.:  V. 

2428^ 

l.OOO 

1 .  '700 

0.143 

anda'ia 

Vfcr, nance 

1.000  ’.OCO 
l.O'OO  0.200 
5.000  1.000 

IX  l!  4.5’a8 

Priorit, 

0.56401 

0,17243 

0.2s 356 

C.I.:  0.’3839 

'  C.R 

592^ 

.  ;; 

915>5 

j 

f'i'ioritv 

l.COO  '.000 

’.000 

0.”77B 

0.143  l.O'OO 

1 .  ooo 

0.11111 

0.143  l.OCO 

1.000 

0.11111 

andaiiax':  3. 

,  0000 

O.dOOOO  2.R.: 

0.  OOOOO 

-aaistiCj 

1  :  3 

Fricrr. , 

1.000  ’.000  01.143 

0.:02’3 

0.14:  l.O'OO  0.143 

0.05540 

'.GOO  ’.000  l.OOC 

.■’.'4136 

affldada.  ".  3.  435’ 

C.i.;  ;.:i';4  C.f.: 

I'll 


APPENDIX  I:  Initial  Survey  Data 


Survivability 

12  3  4  5  6  7  Priority 

1.000  0.125  2.000  0.500  1.000  1.000  0.125  0.05306 

8.000  1.000  7,000  6.000  1.000  4.000  1.000  0.29515 

0.500  0.143  1.000  0.500  0.500  0.500  0.200  0.03938 

2.000  0.167  2.000  1.000  3.000  3.000  0.143  0.10657 

1.000  1.000  2.000  0.333  1.000  2.000  0.200  0.09408 

1.000  0.250  2.000  0.333  0.500  1.000  0.167  0,05140 

8.000  1.000  5.000  7.000  5.000  6.000  1.000  0.36036 

laidadax):  7.7748  C.I.:  0.12913  C.R.:  0.09793 


Prooellants 

1  2  3  Priority 

1.000  2.000  0.500  0.29696 
0.500  1.000  0,333  0.16342 
2.000  3.000  1.000  0.53961 
laada(aax):  3.0092  C.I.:  0.00460  C.fi.:  0,00793 


’laintainability 

1234  Priority 
1.000  0.500  0.500  0.500  0.13567 

2.000  1.000  2.000  1.000  0.339®3 

2.000  0.500  1.000  2.000  0,28734 

2,000  1. 000  0.500  1.000  0.23701 

landalaax)!  4.1855  C.I.:  0.0al84  C.P;.;  0.06871 


flans'j.er  ability 
1  2  3  Pnoritr' 

1.000  0,250  1.000  0,l6it7 
4.000  1.000  4.000  0.666o7 
l.vOO  0.250  l.OOO  0.16667 

landaiaaxl:  3.0000  C.I.:  0.00000  2.f:.:  O.-.vOOO 

'he  CsPsiatencv  ot  this  hierarchy  =  0.v4'?'> 
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APPENDIX  1:  Initial  Survey  Data 


Judge  i  5  AHP  RESULTS  (ORIGINAL  SURVEY) 
TAV  Effectiveness 
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APPENDIX  I:  Initial  Survey  Data 


Survivability 

1  2  3  4  5  6  7  Priority 

1.000  1.000  0.143  0.143  0.200  0.111  0.111  0.01979 

1.000  1.000  0.143  0.143  0.143  0.111  0.111  0.01944 

7.000  7.000  1.000  7,000  7.000  1.000  1.000  0.25279 

:’.000  7.000  0.143  1.000  5.000  0.143  0.111  0.08S05 

5.000  7.000  0.143  0.200  1.000  0.111  0.111  0.0492S 

9.000  9.000  1.000  7.000  9.000  l.MO  1.000  0.27452 

9.000  9.000  1.000  9.000  9.000  1.000  1.000  0.29614 

Uidadaxi:  8.  1462  C.I.:  0.19103  C.R.:  0.14472 


ProDellants 

1  2  3  Priority 

1.000  0.200  0.143  0.07030 
5.000  1.000  5.000  0.67243 
".000  0.200  1.000  C. 25726 

laidadaxli  3.4357  C.I.i  0.21784  C.R.:  0.37559 


»laint3inaii;lit'> 

1234  Priority 
1.000  0.111  0.200  0.333  0.05^34 

5.000  1.000  0,333  0.333  0.155o3 

5.000  3.000  1.000  0.200  0.34600 

3.000  3,000  5.000  1.000  0.49858 

Umciadax):  4.9583  C.:,:  C.332’s  :.P.:  ').3s5'4 


'"aneuverabilitr 
1  2  3  Prisritv 

1.000  ".000  9,000  0.7o0"6 
0,143  1,000  ",C00  0.19115 
0.111  0.143  1.000  0.04805 
lamaaiia;,':  3.32'6  3.1.:  0.16382  C.P.:  0.222-1! 

'tie  Cjnaistenc/  o*  *.tii5  rierarchy  =  0.3C49 
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APPENDIX  I:  Initial  Survey  Data 


Judfe  t  4  AHP  RESULTS  (0RI5INAL  SURVEY) 

TfiV  Effectiveness 

123456  Priority 
1.000  9.000  9.000  9.000  9.000  9.000  0.60031 

0.111  1.000  5.000  1.000  3.000  3.0<)0  0.12656 

0.111  0.200  1.000  l.OOO  0.333  0.333  0.03987 

0.111  1.000  1.000  1.000  3.000  3.0)0  0.10375 

0.111  0.333  3.000  0.333  1.000  5.000  0.08285 

0.111  0.333  3.000  0.333  0.200  1.000  0.04667 

laiida(aa)!):  6.9923  C.I.:  0.  19846  C.R.:  0.16004 
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5. 000 
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1.000 

0.08570 

laiida(«< 

IX):  1 

1.2589 

C.I.: 

5.46555 

C.R.:  0. 

33018 

f'Srforiiance 

1  2  3  Prior: tv 

l.OCO  7.000  0.  143  0.  1911? 

C.143  1.000  v.lll  0.04805 
I’.OOO  9.000  1.000  0.76076 

lamdadav.):  3. 3276  C.I.:  0.16382  C.f;.:  0.28245 


Rists 

1  2  3  Prioritv 

1.000  5.000  5.  000  0.72193 
0.200  l.iXiO  '.000  5.22706 
0.111  0.143  l.OvO  0.05101 
iamdaiia::);  3.2025  C.I.:  0.10423  C.R.:  0.179'1 


.rci sti C3 

1  2  3  Fricrit, 

1.000  0.143  0.111  0.04305 
’.OOO  1.500  i:.14:  0.19115 
'.001  '.yOO  1.010  '/.'sO's 

.aTi'iaiaa; ; :  3, 32’;.  3.1.:  .;;3?2  '3.P.:  ',.22245 
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APPENDIX  1:  Initial  Survey  Data 


Survivability 

1234567  Priority 
1.000  7.000  3.000  7.000  0.333  5.000  0.200  0.19766 

0.143  1.000  0.333  3.000  0.200  1.000  0.200  0.05B89 

0.333  3.000  1.000  3.000  1.000  3.000  0.200  0.09939 

0.143  0.333  0.333  1.000  5.000  3.000  0.143  0.10797 

3.000  5.000  1.000  0.200  1.000  5.000  0.333  0.14531 

0.200  1.000  0.333  0.333  0.200  1.000  0.143  0.02839 

5.000  5.000  5.000  7.000  3.000  7.000  1.000  0.36239 

laiida(U);):  9.7479  C.I.:  0.45799  C.R.:  0.34696 


ProoelUnts 

1  2  3  Priority 

l.OOO  1.000  0.333  0.20000 
1.000  1.000  0.333  0.20000 
3.000  3.000  l.OOC  0.60000 
laadaliax):  3.0000  C.I.:  0.00000  C.R.:  0.00000 


•liintainability 

1234  Prisrity 
1.000  3.000  5.000  1.000  0.42458 

0.333  1.000  3.000  1.000  0.21320 

0.200  0.332  l.OOO  0.233  0.03024 

1.000  l.OOO  2.000  I. MO  0.28198 

UiTida(iax) :  4.1155  2. 1.:  0.038^9  C.R.:  0.04276 


laneuverabilitr 
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lamdadax):  3.1823  C.I.:  0.09138  C.F,.:  O.lS’Sb 

■"he  Consistency  ot  this  hierarchy  =  0.3026 


1-6 


APPENDIX  1:  Initial  Survey  Data 


Judge  t  3  AHP  RESULTS  (ORIGINAL  SURVEY) 


TAV  Effectiveness 
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Priority 
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appendix  I:  Initial  Survey  Data 


Survivability 
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0.09082 
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0.11701 
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1.000 

0.39608 

5.000 

3.IJ00  1.000 

1.000 

0.39608 

laidadiaxl:  4.0328 

C.I.: 

0.01093  C.R.:  0.01215 

naneuverabilitv 

1  2  3 

Prioritv 

1.000  0.500  0.200 

0. 10884 

2.000  1.000  0.167 

0.16258 

5.000  o.OCO  1.000 

0.72858 

laadainax):  3.0856 

C.I.:  0.04238  C.R.:  0.073<’4 

'he  Cjnsietencv  ct  this  hierarchy  =  0.0472 
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APPENDIX  1:  Initial  Survey  Data 


Jud^e  I  2  AHP  RESULTS  (ORIGINAL  SURVEY) 
TAV  ETfectiveness 
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Priority 
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Unida(iidx):  3 
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Logi 
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priority 
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'V.  33-3 
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0.333  1.000 
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laadaliia). ! ;  3. 
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APPENDIX  I:  Initial  Survey  Data 


Survivability 

1234547  Priority 
1.000  0.333  0.333  0.333  3.000  0.200  0.200  0.04006 

3.000  1.000  1.000  1.000  3.000  1.000  0.333  0.14434 

3.000  1.000  1.000  1.000  3.000  1.000  1.000  0.14399 

3.000  1.000  1.000  1.000  3.000  0.333  1.000  0.14434 

0.333  0.333  0.333  0.333  1.000  0.333  0.333  0.04927 

5.000  1.000  1.000  3.000  3.000  1.000  1.000  0.21900 

5.000  3.000  1.000  l.OOO  3.000  1.000  1.000  0.21900 

landa(aax):  7.4316  C.I.:  0.07193  C.R.:  0.05449 


Prooellants 

1  2  3  Priority 

1.000  0.333  0.333  0.14286 
3.000  1.000  1.000  0.42857 
3.000  1.000  1.000  0.42857 
laadaliax);  3.0000  C.I.:  0.00000  C.P.;  0.00000 


flaintainatiilitv 

1234  Priority 
1.000  0.333  0.333  0.333  0.09393 

3.000  1.000  0.333  0.500  0.17414 

3.000  3.000  1.000  3.000  0.48268 

3.000  2.000  0.333  1.000  0.24725 

landamax):  4.2153  C.I.:  0.07175  C.R.:  0.07973 


Maneuverability 
1  2  3  Priaritv 

1.000  0.200  0.333.  0.  10065 
5.. 000  1.000  4.000  0.67381 
3.30C  0.250  1.000  0.22554 
landafitax):  3.0858  C.I.:  0.04288  C.R.:  0.07394 

’'he  Consistency  ot  this  hierarchy  =  0.0209 
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APPENDIX  1:  Initial  Survey  Data 


Judge  I  1  AHP  RESULTS  (ORIGINAL  SURVEY; 
TAV  Effect ivenes'5 
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0.333 

1.000 

0.333 

0,333 

0.05134 
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0.08280 

laida(iiax):  6.3457 

C.I.: 

0.06913  C.ft, 

.:  0.05575 

1 
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Judge  I  7  AHP  RESULTS  (0RI6INAL  SURVEY) 

TfiV  EHectiveness 

1  2  3  4  5  6  Priority 

1.000  9.000  9.000  9.000  9.000  9.000  0.61070 

0.111  1.000  7.000  3.000  7.000  1.000  0.14880 

O.lll  0.143  1.000  0.333  0.333  0.200  0.02581 

0.111  0.333  3.000  1.000  5.000  1.000  0.08300 

0.111  0.143  3.000  0.200  1.000  0.333  0.03740 

0.111  1.000  5.000  1.000  3.000  1.000  0.09429 

laiida(aax):  6.7372  C.I.:  0.14744  C.R.:  0.11390 
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12345678  Priority 
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Judge  I  8  AHP  RESULTS  (ORIS INAL  SURVEY) 
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Survivability 

1234567  Priority 
1.000  0.143  3.000  0.143  0.333  1.000  0.143  0.03863 

7.000  1.000  7.000  1.000  7.000  5.000  5.000  0.36441 

0.333  0.143  1.000  0.143  0.143  0.143  0.143  0.02152 

7.000  1.000  7.000  1.000  3.000  7.000  1.000  0.23819 

3.000  0.143  7.000  0.333  1.000  1.000  0.200  0.07296 

1.000  0.200  7.000  0.143  1.000  1.000  0.200  0.05997 
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5.000  0.333  1.000  0.27895 
laada(aax)!  3.0649  C.I.:  0.03244  C.R.:  0.05594 


Maneuverability 
1  2  3  Priority 

1.000  5.000  ^000  0.^014? 

0.2C0  1.000  ■'.000  0.23991 
0.143  0.143  1.000  0.05360 
laadaiiax):  3.2948  C.I.:  0.14739  C.R.:  0.;5412 

fhe  Consiitericv  at  this  hierarchy  =  0.1600 


I~16 


APPENDIX  I:  Initial  Survey  Data 


Judge  t  9  AHf>  RESULTS  (ORIBINAL  SURVEY) 
TfiV  Effectiveness 


I 

^  r 

A  J 

4  5 

6 

Priority 

1.000 

0.143  0.333 

0.200  0.333 

0.333 

0.03973 

7.000 

1.000  3.000 

3.000  1.000 

3.000 

0.32302 

3.000 

0.333  1.000 

0.333  0.333 

0.333 

0.07399 

5.000 

0.333  3.000 

1.000  3.000 

4.000 

0.26985 

3.000 

1.000  3.000 

0.333  1.000 

3.000 

0. 18997 

3.000 

0.333  3.000 

0.250  0.333 

1.000 

0.10344 

laiida(eax):  6.6031 

C.I.:  0.1206 

3  C.R.:  0.09728 

1 

Requireients 

'>  T 

^  w' 

4  5 

6 

7  3 

Priority 

1.000 

5.000  7.000 

3.000  3.000 

7.000 

3.000  0.333 

0.24621 

o.:oo 

1.000  7.000 

1.000  3.000 

7.000 

3.000  0.333 

0.13203 

0.143 

0.143  l.OOO 

0.143  0.200 

0.133 

0.200  0.14! 

0.01926 

0.333 

1.000  7.000 

1.000  3.000 

5.000 

3.000  1.000 

0.15014 

0.33! 

0.333  5.000 

0.333  l.OOO 

3.000 

3. COO  0.300 

0.07608 

0.143 

0.143  3.000 

0.2')0  0.333 

1.000 

0.200  0.200 

0.02964 

0.333 

0.33!  5.000 

0.333  0.333 

5. 000 

l.OOO  0.200 

0,06300 

3.000 

3.000  7.000 

1.00  0  5.000 

5.000 

5.000  1.000 

0.28364 

laida(iax):  6.9796 

C.I.:  0.13954  C.R, 

0,0®92- 

Perforsance 

1.000  3.000  0.143 

’.000  7.000  1.000 
laigdaliax);  r.l!5Q 

Pnaritt 

0.15787 

0.07590 

0.76623 

C.I.:  0.06781  C.R, 

.:  0.11651 

r*.  1  5  ^  S 

1  •»  •* 

1.000  1.000  0.333 
1.000  1.000  0.333 
3.000  3.000  1,000 
lamdai.iax):  J.OO'jO 

frionti 

0. 20000 

0.20000 

0,60000 

C.I.:  .i.OOOOo  C.R 

. :  O.i'H'ii'ino 

Logistics 

i  T  -* 

i  a.  V 

Pnority 

000  3.000  3.0C0 

0.60000 
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Survivability 

1234567  Priority 
1.000  0.333  0.200  0.200  3.000  0.200  0.200  0.04766 

3.000  1.000  0.333  0.333  5.000  0.333  0.333  0.09342 

5.000  3.000  1.000  I. 000  3.000  I. 000  1.000  0.19353 

5.000  3.000  l.OOO  1.000  3.000  l.OOO  1.000  0.19353 

0.333  0.200  0.333  0.333  1.000  0.200  0.200  0.03842 

5.000  3.000  1.000  1.000  5.000  1.000  3.000  0.25204 

5.000  3.000  1.000  1.000  5.000  0.333  1.000  0.18140 

la«la(Mx):  7.5146  C.I.!  0.08576  C.R.:  0.06497 
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1  2  3  Priority 

1.000  1.000  0.143  0.11111 
1.000  1.000  0.143  o.inii 
7.000  7.000  1.000  0.77778 
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Judge  «  10  AHP  RESULTS  (ORIS INAL  SURVEV) 

TAV  Effectiveness 

123456  Priority 
l.OOO  3.000  3.000  3.000  3.000  3.000  0.34463 

0.333  1.000  3.000  1.000  0.333  0.333  0.09981 

0.333  0.333  1.000  3.000  0.333  0.333  0.08445 

0.333  1.000  0.333  1.000  0.333  0.333  0.06622 

0.333  3.000  3.000  3.000  1.000  0.333  0.16582 

0.333  3.000  3.000  3.000  3.000  1.000  0.23906 

laodadax):  6.7049  C.I.:  0.14098  C.R.:  0.11370 


Reauiresents 


1 

n 

7 

4 

J 

6 

T 

/ 

8 

Priority 

1.000 

1.000 

3.000 

1.000 

3.000 

3.000 

3.000 

1.000 

0.17917 

1.000 

1.000 

1.000 

0. 063 

1.000 

3.000 

l.OOO 

1.000 

0.10828 

0.333 

l.OOO 

1.000 

0.333 

1.000 

3.000 

0.333 

0.333 

0.07379 

1.000 

3.000 

3.000 

1.000 

5. 000 

5.0()0 

3.000 

1.000 

0.23431 

(‘i  ■f’r 

1.000 

1.000 

0.200 

1.000 

l.OOO 

3.000 

A  T'*7 

V* 

0.08176 

0.333 

0.333 

0. 333 

0.200 

1.000 

l.OOO 

1.000 

0.333 

0.05051 

0.333 

1.000 

3.000 

0.333 

0.333 

1.000 

1.000 

0.200 

0.0:’566 

1.000 

1.000 

3.000 

1.000 

3.000 

3.000 

5.000 

1.000 

0.19652 

laadaleaxl :  6 

.7258 

C.I.: 

0.1036 

8  C.R, 

0.07353 

Per^oraance 

1  2  3  Priority 

3.000  1.000  0.200  0.20212 
5.000  5.000  1.000  0.70071 

laaca(iax):  3.1256  C.I.:  0.06701  C.R.:  0.11691 


Pi  3  5 

1  2  3  '''•loritv 

l.OCO  0.50  0  0.500  0.20000 
2.000  l.OOU  1.000  0.40000 
2.000  1.000  1.000  0.40000 

laadaiiax) ;  3.0000  2.  1.:  O.'.'OOOO  C.P.:  O.'.'OOOo 


Lcqistics 

1  2  3  ►’’icrit. 

I.jOO  3.000  0.111  0.12952 
0.333  1.000  0.111  0.0i227 
=  .000  ‘>.000  l.OOi)  0.80822 

laiTdaiia;- ) :  3.1  356  2. 1.:  2.P.:  O.lli^’l 
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APPENDIX  Is  Initial  Survey  Data 


1 

Survivability 

2  3 

4 

5 

6 

7 

Priority 

1.000 

1.000 

0.200 

1.000 

0.333 

0.200 

0.200 

0.05306 

1.000 

1.000 

0.200 

0.333 

0.333 

0.333 

0.333 

0.04107 

5.000 

5.000 

1.000 

5.000 

5.000 

5.000 

5.000 

0.40561 

1.000 

3.000 

0.200 

1.000 

5.000 

3.000 

3.000 

0.19065 

3.000 

3.000 

0.200 

0.200 

1.000 

1.000 

3.000 

0.10537 

5.000 

3.000 

0.200 

0.333 

1.000 

1.000 

3.000 

0.12116 

5.000 

3.000 

0.200 

0.333 

0.333 

0.333 

1.000 

0.08308 

laedatuxl:  8, 

\  jL!  ^ 

C.I.: 

0.2212 

1  C.R, 

1  •  0* 16758 

Frooellants 

1 

1  -j 

Priority 

1.000 

7.000  3.000 

0.64912 

0.143 

1.000  0.200 

0.07193 

0.333 

5.000  1.000 

0.27895 

laidatiaxi:  3.0649 

C.I.!  0.03244  C.P.:  0.05594 

riaintainability 

1  2  3 

4 

Priority 

1.000  7.000  7.000 

5.000 

0.s4484 

0.143  1.000  5.000 

3.000 

0.15210 

0.143  0.200  1.000 

0.200 

0.04425 

0.200  0.333  5.000 

1.000 

0.11861 

lamdaliax):  4.4868 

C.I.: 

0.1t22e  C.ft.:  0.18031 

I'tansu.erabil 

:ty 

Priority 

1.000  0.200  }. 

•rw  • 

0.11397 

5.000  1.000  1. 

000 

0.48064 

3.0CO  1.000  1. 

000 

0.40539 

laiKlaiiaxi:  3.02 

:9i 

C.I.:  0. 01453  C.P.:  0.02506 

'’he  Consistency  ci  tiiersrc.ny  =  0.1050 


APPENDIX  I:  Initial  Survey  Data 


Judge  «  11  AHP  RESULTS  (ORIGINAL  SURVEY) 


TAV  Effectiveness 

1  2  3 

4 

5 

b 

Priority 

1.000 

7.000 

7.001) 

7.000 

7.000 

7.000 

0.49290 

0.143 

1. 000 

7.000 

7.000 

7.000 

7.000 

0.25491 

0.143 

0.143 

1.000 

0.200 

0.143 

0.143 

0.01970 

0.143 

0.143 

5.000 

1.000 

0.143 

0.143 

0.03248 

0.143 

0.143 

7.000 

7.000 

l.OOO 

7. 000 

0.13183 

0.143 

0.143 

7.000 

7.000 

0.143 

1.000 

0.06817 

laadadax):  8, 

,2015 

C.I.: 

0.44031 

,  C.R 

.:  0.355<)9 

1 

Reguireients 

4  ^ 

4 

5 

6 

7 

8 

Priority 

1.000 

7.01)0 

9.000 

7.000 

9.000 

9.000 

9  000 

7.000 

0.32068 

0.143 

1.000 

9.000 

0.143 

9.000 

9.0IX' 

9.000 

5.000 

0.16629 

0.111 

0.111 

1.000 

9.000 

7.000 

1.000 

0.143 

0.143 

0.11795 

0.143 

7. 000 

0.111 

1.000 

7.000 

9.000 

7.000 

7.000 

0. 17245 

0.111 

O.lll 

0.143 

0.143 

1.000 

7.000 

7.000 

0.142 

0.03957 

0.111 

0.111 

1.000 

0.111 

0. 143 

1.000 

0.143 

0.143 

0.01357 

0.111 

0.111 

7.000 

0.143 

0. 143 

7.000 

1.000 

0.143 

0,06265 

0.143 

0.200 

7.000 

0.143 

7.1X10 

7.000 

7.000 

1.000 

0.10683 

ia(iida(eax);  1 

7.2861 

C.I.: 

1.3:( 

b59  C.R.:  0.94084 

Per<oriance 

1 

i 

■-1  ^ 

to  ‘J 

Prioriti 

1,000 

7.000  7,000 

0,’4186 

0.143 

1.000  ’.000 

0.20273 

0.143 

0.143  1,000 

0.05540 

laflda(iax)!  3.435'’ 

C.I.:  O.jrSA  C.R.:  0.37559 

Risks 

4  ~ 

«  to  W 

fTioriti 

1,000  0.200  0.  143 

0.06292 

5.00';’  1.000  0.143 

0.18397 

’.'lOO  '.COO  1.000 

0.75311 

aiidaiia;:) :  3.2°48 

C.I.:  0.14’:9  C.P.: 

0.25412 

uoqi sties 

Priori  tv 

l.OCO  ’.UOO  3.0100 

0,13299 

0.143  1.000  0.143 

0 . 06270 

0.333  '.000  1.000 

0.30431 

sfDua.Aax,'* 

2 . . , :  .  V  6 ,  8 1 

C.lli^l 
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APPENDIX  I:  Initial  Survey  Data 


Survivability 

1234567  Priority 
l.OOO  O.lll  0.143  0.143  0.143  0.143  0.143  0.01580 

9.000  1.000  7.000  5.000  7.000  7.000  7.000  0.44906 

7.000  0.143  1.000  0.143  5.000  5.000  0.143  0.07343 

7.000  0.200  7.000  1.000  7.000  5.000  0.200  0.14879 

7.000  0.143  0.200  0.143  1.000  l.OOO  0.143  0.0351'’ 

7.000  0.  143  0.:M0  0.200  1.000  1.000  0.143  0.03613 

7.000  0.143  7.000  5.000  7.000  7.000  1.000  0.24161 

laadadax):  8.9959  C.I.:  0.33265  C.R.:  0.25201 


Propellants 

1  2  3  Priority 

1.000  9.000  7.000  0.76076 
0.111  1.000  0.143  0.04805 
0.143  7.000  1.000  0.1911? 
laadaiU!!):  3.3;7s  C.I.:  0.  16382  '..R.:  0.28245 


fliintainacility 

1  2  3  4  Priority 

1.000  0.143  0.111  0.200  0.03372 

'  I  l.  j'.‘0  1 4 ^  5..VV  ^.^0401 

9.000  7.000  l.OOO  ’.000  ■;.6S2’0 

5.000  0.200  0.111  1.000  0,07957 

lamdadax):  4.5353  C.I.:  0.19511  C.R.;  0.21679 


Maneuverability 


Frioriti 


l.OOO 

I'.OOO 

^O'TO 

0.,'’4IS6 

0,143 

1.000 

'.000 

0.20273 

0.143 

0. 143 

1 . 000 

0.05540 

andai.iax) :  3. 

,435' 

3. 1.:  i 

'he  Consistencr  ct  tris  hierarchy  =  0.6946 
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APPENDIX  I:  Initial  Survey  Data 


Judge  I  12  AHP  RESULTS  (ORIGINAL  SURVEY) 
TAV  ETfectiveness 


I 

n 

L 

W 

4 

5 

6 

Priority 

1.000 

7.000 

1.000 

5.000 

5.000 

5.000 

0.35841 

0.143 

1.000 

1.000 

5.000 

3.000 

3.000 

0.16746 

1.000 

1.000 

1.000 

5.000 

1.000 

0.200 

0.14129 

0.200 

0.200 

0.200 

1.000 

3.000 

3.000 

0.10010 

0.200 

0.333 

1.000 

0.333 

1.000 

7.000 

0.12616 

0.200 

0.333 

5.000 

0.333 

0.143 

1.000 

0. 10657 

landadax):  ' 

7.3079 

C.I.: 

0.6615 

7  C.R, 

.:  0.53353 

1 

Reouireients 

7 

i.  J 

4  5 

6 

i 

3 

Priority 

1.000 

7.000 

7.000 

5.000  0.143 

0.200 

0.200 

5.000 

0.13131 

0.143 

1.000 

1.000 

0.200  0.143 

0.200 

0.200 

5.000 

0,05143 

0.143 

1.000 

1.000 

0.200  0.143 

0.333 

0.143 

5.000 

0.05147 

0.200 

5. 000 

5.000 

1.000  0,143 

3,000 

0.143 

5.000 

0.09954 

7.000 

7.000 

7.000 

7.000  1.000 

5.000 

5.000 

3.000 

0.28512 

5.000 

5. 000 

3.000 

0.333  0.200 

1 . 000 

0.143 

0.200 

0.08547 

5.000 

5.000 

7.000 

7.000  0.200 

7.000 

1.000 

0.200 

0.18753 

0.200 

0.200 

0.200 

0.200  0,333 

5.000 

5.000 

1.000 

0.10813 

landatiaxl;  15.1192 

C.I.!  1.017 

'03  C. 

R.:  0. 

72129 

Perforinanca 

1  : 

1.000  ■’.000  1.000 
0.143  1.000  0.200 
1.000  5.000  I. 000 
laiiida(iax!;  3.0126 

Priority 

0.48692 

0.0'732 

0.43526 

C.I.:  0.00630  C.P 

.:  0.01086 

*<1515 

1 

a 

2  3  Priority 

l.OCO  0, 

20C  0.143  0.07782 

5.700  1, 

,000  1.000  0.43526 

’.000  1. 

000  1.000  0.48692 

amdaiiax 

i:  3.0i:s  C.I.:  0.00637 

0. Cl 036 

.C’G  1  3  V  1  C3 

F'^ioritv 

l.OCO  0.143  ,:i,i43 

0.i>5540 

’.70j  1.700  C.143 

0.202 ’3 

’.OcO  ’,000  1,000 

0.’4136 

iiDda' 'a, ; :  3.435’ 

C.I.:  0.2:’64  2.9.: 

0.3’:5^ 
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APPENDIX  I:  Initial  Survey  Data 


Survivability 

1234547  Priority 
1.000  3.000  0.143  1.000  3.000  0.333  3.000  0.14849 

0.333  1.000  5.000  7.000  3.000  3.000  3.000  0.25399 

7.000  0.200  1.000  1.000  3.000  3.000  0.200  0.17498 

1.000  0.143  1.000  1.000  3.000  3.000  0.200  0.07845 

0.333  0.333  0.333  0.333  1.000  1.000  0.143  0.03454 

3.000  0.333  0.333  0.333  1.000  1.000  0.200  0.07788 

0.333  0.333  5.000  5.000  7.000  5.000  1.000  0.21127 

laida  («.'<):  10.4547  C.I.:  0.60944  C.R.:  0.44171 


Prooellants 

1  2  3  Priority 

1.000  9.000  5.000  0.75343 
0.111  l.OCO  5.000  0.17413 
0.200  0.200  1.000  C. 07244 
laadalnax);  3.5408  C.l.:  0.28042  C.P.:  0.48348 


'taintairaoiltt. 

1234  “ri:r:tv 
1.000  5.000  0.200  0.200  0.11383 

0.200  1.000  0.200  0.143  0.04719 

5.000  5.000  1.000  0.333  v.297a8 

5.000  7.0')0  3.00'J  1.000  0.53430 

iamaatia*;:  4.3800  C.I.:  0.12«68  C.F.:  v. 14409 


•areuieraDil'.t'f 
1  2  3  Priority 

1.000  ’.000  3.0CO  0.64912 
0.143  1.000  0.200  0.07193 
0.333  5.000  l.OCO  0.27395 

lanifla'iia*):  3.0549  C.I.:  C. 03244  C.R.:  0.05594 


'7e  Consistency  of  this  merarchy  =  0.5034 


APPENDIX  I:  Initial  Survey  Data 


6NMtric  Ntan  for  12  Origiul  Surveys 


TAV  EfiKtivencss 

1  2  3 

4 

5 

6 

Priority 

1.000 

1.754 

2.063 

3.605 

2.928 

2.199 

0.31191 

0.570 

1.000 

2.367 

2.945 

2.160 

2.205 

0,24071 

0.485 

0,423 

1.000 

1.615 

0.670 

0.679 

0.10957 

0.27? 

0.340 

0.619 

1.000 

1.178 

0.689 

0.09067 

0.342 

0.463 

1.492 

0.349 

1.000 

1.629 

0. 12673 

0.455 

0.453 

1.474 

1.452 

0.614 

1.000 

0.12040 

laida(iiax):  1 

b.  1652 

C.I.: 

0.03304  C.R 

.:  0.02665 

1 

Reauireients 

n  •? 

4 

5 

6 

3 

Priority 

1.000 

2.234 

4.620 

2.315 

1.894 

4.453 

1.859 

0.26178 

0.448 

1,000 

3.319 

1.373 

1.382 

3.362 

1.725 

2.016 

0. 17003 

0.216 

0.301 

1.000 

0.632 

0.905 

1.538 

0.396 

0.542 

0.06429 

0.355 

0.723 

1.582 

l.txlO 

1.253 

3. 637 

1.U37 

1.000 

0.11892 

0.528 

0.724 

1.105 

0.738 

1.000 

1.90a 

2.085 

0.850 

0.11636 

0.225 

0.297 

0,650 

0.275 

0.500 

! .  000 

0.371 

0.399 

0.04549 

0.449 

0.580 

2.526 

0.965 

0.430 

c.D^^a 

1.000 

0.S25 

0.1C649 

0.538 

0.496 

1.844 

0.'991 

l.Tb 

2.504 

1.212 

1.000 

0.11664 

andadax);  8. 

2133 

C.!.: 

0.03047  C.R, 

0.0: 

2161 

Psrforiance 

1 

'I 

Prion  t.' 

l.OOO 

2. 7'’9 

4  4^15 

0, 

4’S97 

0.357 

1.000 

0 .  fiO  1 

[<. 

,I35’0 

0.760 

1.664 

l.OOO 

0. 

andaliax):  3. 

,006:' 

C. 

Li  0.00335  C.R.:  0.0057B 

“^isks 

1  3 

P'-lOntr 

l.OOO  '.'.566 

0.23:66 

l.M'O  1.000  0.956 

0.38235 

1.562  1.043  l.vOO 

a/idanax':  .M'l’l: 

5.;.:  o.,'evt: 

:.P.: 

O.’.'OIId 

Legist;  ;= 

i.wu  1.;:;  0.310 
U.383  l.'M  0.33' 
;.3!8  1.  403  l.OOv 
laiKlania*::  3.>.'1l4 


-’■IC'lt. 

e.-OlM 


';.13',33 
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APPENDIX  I;  Initial  Survey  Data 


Survivability 

1234567  Priority 
1,000  0.496  0.353  0.476  0.632  0.367  0.235  0,05896 

2.017  1.000  0.959  1.678  1.733  0.353  0.590  0.14988 

2.835  1.043  1.000  1.208  2.231  1.125  0.480  0.15592 

2.100  0.596  0.828  1.000  2.779  1.168  0.375  0.13540 

1.582  0.577  0.448  0.360  1.000  0.^35  0.2‘’3  C. 07887 

2.722  1.172  0.388  0.856  1.361  I. 000  0.500  0.13732 

4.2*63  1.696  2.083  2.665  3.411  1.999  1.000  0.28364 

!a#(la(Mx):  7.1164  C.I.:  0.01940  C.R.;  0.01470 


Prooellants 

1  2  3  Priority 

l.OOO  2.025  0.7S6  0.37577 
0.494  1.  000  0.636  0.21879 
1.2’2  1.571  1.000  0.40544 
laBda (lax) :  3.0272  C.I.i  0.01362  C.P.:  0. 02348 


liintairaSilitv 

1  2  3  4  Pricrit/ 

l.lX'O  0.470  V.490  0.  1. ''24 

2.123  1.000  0.  719  1.04«  0.265:3 
:.  667  1.351  i.'OOO  1.066  0.33335 

2.042  O'. 553  O.-rs  I.v'jO  y.:-4iB 

lisida^aao:  4..'0'95  C.i.!  0.ijC3ltj  O.P.:  >.'.0'.3:'4 


niararchv  =  0.'j1j4 


APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Survivability 


1 

3 

♦ 

5 

6  7 

Priority 

l.OOO 

1.000 

0.111 

1.000 

1.000 

l.OOO  1.000 

0.06320 

1.000 

1.000 

0.111 

5.000 

7.000 

1.000  1.000 

0.12804 

9,000 

9.000 

1.000 

9.000 

9.000 

9.000  9.000 

0.56878 

1.000 

0.200 

O.lll 

1.000 

0.333 

1.000  1.000 

0.04548 

1.000 

0.143 

0.111 

3.000 

1.000 

1.000  0.333 

0.05423 

1.000 

1.000 

0,111 

1.000 

1.000 

1.000  1.000 

0.06320 

1.000 

l.OilO 

0.111 

1.000 

3.000 

1.000  1.000 

0.07706 

aidallax):  7 

.3232 

C.I,: 

0.1372 

1  C.R.:  0.10394 

Prooellanta 

j 

n  ^ 

te  J 

Priority 

1.000 

0.200  0.200 

0.09091 

5.000 

1.000  1.000 

0.45455 

5.000 

1.000  1.000 

0.45455 

laadaHa 

3.  0000 

2. I.:  0.00000 

C.R.: 

0.00000 

liintainaDilitv 

1  2  3 

4 

Priority 

1.000 

C.20C  0.200 

0.200 

0, 06250 

5.000 

1.000  1.  000 

1.000 

0.31250 

5.000 

1 .  lOOO  1 , 000 

1 . 000 

0.31250 

5.000 

1.000  l.OOO 

1.000 

0.31250 

lamdaiia 

x):  4.0000 

C.I.: 

0.00000  C.R.!  0.00000 

lanejterabilit. 

P'^iDritv 

I.;';';  1.000  5.000 

'0.  45455 

■  .  1111)  •. .  ili'ii'i  .  ;■)(>' 

0.45455 

,200  ;.2v0  1.000 

0.090'’ 1 

limca  ia;  :  O.i'OOl 

2,1.:  0.00000  2.F:. 

:  0 . 00000 

'“’s  l:'5i5tencf  ;♦  ‘/i5  ni5''arc'iv  =  0.0510 
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APPENDIX  J:  AHP  Fall aw— Up  Survey  Data 


Judge  I  7  AHP  RESILTS 
TAV  Effectiveness 

1  2  3  4  5  6  Priority 

1.000  9.000  9.000  9.000  9.000  9.000  0.61070 

O.in  1.000  7.000  3.000  7.000  1.000  0.14880 

O.llt  0.143  1.000  0.333  0.333  0.200  0.02581 

0.111  0.333  3.000  1.000  5.000  1.000  0.08300 

O.lll  0.143  3.000  0.200  1.000  0.333  0.03740 

0.111  1.000  5.000  1.000  3.000  1.000  0.09429 

laeda(iax):  6.7372  C.I.:  0.14744  C.R.;  0.11890 


Reauiresents 

I  2  7  4  j  6  ’  3  friarity 

1.000  5.000  ”.000  5.000  5.000  ?.0C0  9.000  3.000  0.41035 

0.200  1.000  5.000  1.000  1.000  5.000  5.000  1.000  0.12081 

0.111  0.200  1.000  O.lll  0.332  1.000  I. 000  1.000  0.03642 

0.200  1.000  9.000  1.000  3.000  9.000  5.000  1.000  0.17448 

0.200  l.'WO  3.000  0.333  1.000  7.000  5.000  1.000  0.10501 

0.111  0.200  1,000  0.111  0,143  1.000  1.000  0.333  0.02656 

O.lll  0.200  1.000  0.200  0.200  l.OOO  1.000  0.333  0.02906 

0.333  1.000  1.000  l.OOO  1.000  3.000  3.000  l.OOO  0.09732 

laiKladax):  8.6076  C.I.:  0.03680  C.R.:  0.06156 


Perforeance 

1  2  3  Priority 

l.OOO  3.00C  3,000  0.60000 
0.333  l.OOij  1.000  0,20000 
0.333  l.OOO  1,000  0.20000 
laida(eax):  3.0000  C.I.:  0.00000  C.R,:  0.00000 


Risks 

1  2  3  Priority 

:.}00  1.000  1.000  0.33333 
1.000  l.OOO  1.000  0.33333 
;.}00  1.000  1.000  0.33333 

!aMa(iax);  3,00'00  C.I.:  0.00000  C.R.:  0.00000 


^agistics 

1  2  3  Frioritv 

1.000  5.000  1,000  0.45455 
0.200  1.000  0.2'»  0.09091 
1.100  5.000  1.000  0.45455 

laeCa‘i»aO:  3.0000  C.I.:  0,00000  3.R.:  0.00000 
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APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Survivability 

1234567  Priority 
1.000  0.143  0.143  1.000  0.143  0.143  0.143  0.02452 

7.000  1.000  1.000  5.000  1.000  0.143  0.111  0.09498 

7.000  1.000  1.000  1.000  1.000  0.143  0.143  0.07408 

1.000  0.200  I. 000  1.000  1.000  0.143  0.111  0.04590 

7.000  1.000  1.000  1.000  1.000  1.000  1.000  0.13842 

7.000  7.000  7.000  7.000  1.000  1.000  1.000  0.29405 

7.000  9.000  7.00t5  9.000  1.000  1.000  1.000  0.32805 

laidadas):  8.2872  C.I.:  0.21453  C.R.:  0.16252 


Propellants 

1  2  3  Priority 

1.000  7.000  7.000  0.76623 
0.143  1.000  0.333  0.07590 
0.143  3.000  1.000  0.15787 

laadaliaxi!  3.1356  C.I.:  0.06781  C.P.:  0.11691 


flaintainabilitv 

1  2  3  4  Ft:  an  tv 

1.000  0.143  0.333  1.000  0.07521 

■'.000  1.000  5.  111  5.000  0.23447 

3 . 000  ^ .  000  1 . 0'X'  3 . 000  0 .  a  1 253 

1.000  0.200  0.333  1.000  0.077’? 

laiKjaiaax):  5.1944  C.I.:  0.39815  C.R.:  0.44238 


“lar.eu.er  ability 
1  2  3  Priari:* 

1.000  ’.  000  ’.000  0.  777’S 
0.143  l.'OOC  1.000  0.11111 
0.143  l.OOO  1.000  0.11111 

laadalaan:  3.00^00  C.I.:  0.00000  C.F:.:  0.00000 

’he  Ccnsisrenc.  at  this  hierarchy  =  0.3611 
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appendix  J:  AHP  Follow-Up  Survey  Data 


Judge  ♦  6  AHP  RESULTS 
TAV  Effectiveness 

12  3  4  5  6  Priority 

1.000  0.333  7.000  7.000  7.000  0.143  0.18057 

3.000  l.OOO  7.000  5.000  7.000  1.000  0.27843 

0.143  0.143  1.000  0.143  5.000  0.143  0.0438? 

0.143  0.2CO  7.000  1.000  1.000  0.143  0.07269 

0.143  0.143  0.200  1.000  1.000  0.143  0.03104 

7.000  1.000  7.000  7. MO  7.000  1.000  0.39339 

laiDda(iidx):  7.5476  C.I.:  0.30951  C.R.:  0.24961 


Reaui resents 


1 

T 

4 

J 

a 

8 

Priority 

1.000 

7.000 

7 . 000 

7.000 

7.000 

9.000 

0.200 

1.000 

0.23684 

0.143 

1.000 

5.000 

5.000 

0.143 

7.000 

0.143 

1.000 

0.07532 

0.143 

0.200 

1.000 

0,200 

1.000 

5, 000 

0.143 

0.333 

0,03570 

0.143 

0.200 

5.000 

1.000 

0.200 

5.000 

0.143 

0.3:3 

0.04417 

0.143 

7.0M 

1.000 

5.000 

1.000 

5.000 

0.200 

1.000 

0.11454 

0.111 

0. 143 

0.200 

0.200 

0.200 

1.000 

0.111 

0.143 

0.01432 

5.000 

7.000 

'.000 

'.000 

5.000 

».  000 

1.000 

7.000 

0.33995 

1.000 

l.OOC 

3.000 

3.000 

l.OOO 

7.000 

0.143 

1.000 

0.069! a 

la«da(>iax) : 

«A 

I  V  •  •' 

C.i.: 

0.317 

’47  C. 

R.:  0. 

22516 

?erfor*ance 

1  2  3  Priantf 

1.000  1.000  ■'.000  0.5a40l 
1.000  1.000  C.200  0,17243 
:.113  5.300  1,000  0.26356 
laradaiias):  4.5763  C.I.:  0.78339  C.R.:  1.35?:^ 


sis 

2  3  Prion  Cl 

MOO  ’.oco  '.:’:'7’B 

i.:''0'  i.ioo'  0.11111 

1  .  . 1 .  :’}K‘  .  i  1 1 11 

Uii'Ca'ia?  ;  3.0000  3.  1.:  v.OOOOO  C.R.:  J.'A-oiO'.- 

.CClStllS 

..:i3 

'.?0;  1.100 
lamca  13.  :  3.<3;' 
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APPENDIX  Js  AHP  Fallow-Up  Survey  Data 


Survivability 


1 

4 

5 

6 

/ 

Priority 

1.000 

0.125 

2.000 

0.500 

1.000 

1.000 

0.125 

0.05306 

B.OOO 

1.000 

7.000 

6.000 

1.000 

4.000 

1.000 

0.29515 

0.500 

0,143 

1,000 

0.500 

0.500 

0,500 

0.200 

0.03938 

:.ooo 

O.lsT 

:.oco 

l.i'OO 

7.000 

7.000 

0. 147 

0.10657 

1.000 

l.COO 

2.000 

0. 772 

1.000 

:.;)oo 

O'.ZOO 

0.09408 

1.000 

0.250 

2.  OOC 

0.500 

l.OOO 

V  •  i  3 . 

0.05140 

8.000 

1.000 

5.00C 

’.000 

5.000 

i.OO'.' 

1.000 

7.76076 

laridaliax; :  7 

.7748 

^  ^  • 

L.  « 

.A  <  < 

V.  ,4.T  j 

7  "  ' 

1  •  ^ 

9'93 

F'-cael 

lar.ti 

-■■iir;*., 

0.507 

0.2’6'- 

2.c:0  7.700 
asida  ''nas :  7 

A*' 

0.16742 

0.57=61 

7.o:i60 

r  r.  ^ 

*j::‘,a;'al: 

•  '  ,  i 

t  ^ 

«  AA’*  t'  CAf- 

•  «  .  .  *.  I  •  .  • 

S  ••  •••  A  .  •»  C  i  ^ 

2.}00  1.000 

3.000 

l.OOO  0.77”8 

2.000  0.500 

1 , 300 

2.000  7.23’7-l 

2.000  1.000 

0.500 

1.000  0.23701 

lamdaiflax) :  4. 

4  cec 
iOvJ 

C.I.:  0.06134  C.H.:  0.06871 

."laneuver  ability 

«  A  -* 

i  V 

^risritv 

1.000  0.250  1,000 

0.16667 

4.000  1.000  4.000 

0.66667 

1.000  C.25C  l.COO 

C.  16667 

lamdaiaas. :  7.y000 

C.I.:  O.C'OOOO  C.R.:  0.00000 

’'ne  Cjnsistsncv  :t  *.^l3  'lerjrih',  = 
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APPENDIX. Js  AHP  Follow-Up  Survey  Data 


Judge  »  5  AHP  RESULTS 
TAV  Effectiveness 


1 

n 

y 

4 

5 

6 

Priority 

l.OOO 

0.500 

0.500 

3.000 

2.000 

1.000 

0. Ib62b 

:.ooo 

1.000 

1.000 

2.000 

2.000 

2.000 

0.23371 

2.000 

1.000 

1.000 

3.000 

2.000 

3.000 

0.26772 

0.333 

0.500 

0.333 

1.000 

2.000 

0.500 

0.10131 

0.500 

0.500 

0.500 

0.500 

1.000 

2.000 

0.11420 

l.OOO 

0.500 

0. 333 

2.000 

0.500 

1.000 

0.1167? 

landaiiax) :  i 

i).4l21 

C.I.: 

0.08243  C.R, 

0.06648 

1 

fesjireeents 

4 

5 

3 

7 

a 

Priority 

1.000 

0.333 

2.000 

3.000 

0.250 

t.n 

<::• 

4.000 

1.000 

0.12':’37 

3.000 

1.000 

2.000 

5.000 

^  .  .‘V 

5,000 

4.000 

4.000 

0.21474 

0.  jOO 

0.500 

1.000 

5.000 

0.500 

5.000 

4.000 

2.000 

0.1400? 

0.333 

0.200 

0.200 

l.OOO 

0,200 

2.000 

2.000 

2.000 

0.05S43 

4.00C 

3.000 

2.000 

5.000 

1.000 

6.  COO 

6.000 

5.000 

0.32380 

o.:co 

'■  "‘'"'ir 

lOO 

0.500 

1.000 

2.000 

0.500 

0.03649 

0.250 

0.250 

0.250 

0.50C 

O.lo’ 

0.500 

1.000 

0.500 

C. 03332 

1.000 

0.250 

0. 500 

0.500 

0.200 

2.  OOO 

2.000 

l.OOO 

0.00376 

.aiiaa'i 

u?.;:  S 

1  w  *  w  . 

''  ^  . 

y  •  i  i  • 

O.OW  C.R 

.:  O.l 

)i4h 

^ir^srunce 

PriD'-it. 

: .  ’)co 

3.  JOO 

1.000 

0.4*343 

1.000 

0.500 

V. 16-20 

2,70; 

1.000 

O',  38737 

jiDda ' 

3, 

,oie3 

C.I.:  O'. 00-15  C.R.: 

O.Olt'” 

-151(3 

1  ^  - 

.f'riirit, 

l.!0O  3.0,'’.  2.^C"0' 

0.54S95 

0.33:  l.O'V: 

0.20984 

.!.5';'3  ;.ooo 

0.24021 

laadai.ia?) ;  3.0153 

:..o’i5  :.f.: 

■  i 

P-;crit, 

;'..;342 

2. ! !.v;0  O.w!’.' 

0.20-4* 

3,.'  0  2.  OvO'  1,000 

0 aada la ■:  ;  3.U0'?2 

C.  i,  i  , }  j^z'.  : 
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APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Survivability 

1  2  3  4  5  6  7  Priority 

1.000  1.000  0.143  0.143  0.200  0.111  0.111  0.01973 

1.000  1.000  0.143  0.143  0.143  0.111  0.111  0.01915 

7.000  7.000  1.000  7.000  7.000  1.000  1.000  0.25006 

7.000  7.000  0.143  1.000  0.200  0.143  0.111  0.05476 

5.000  7.000  0.143  5.000  l.OOO  O.lll  0.111  0.08356 

9.000  9.000  1.000  7.000  9.000  1.000  1.000  0.27973 

9.000  9.000  1.00(7  9.000  9.000  1.000  1.000  0.29301 

landadax):  8.2509  C.I.:  0.20B48  C.R.:  0.15794 


Prooellants 

1  2  3  Priority 

1.000  0.200  0.143  0.07030 
5.000  1.000  5.000  0.67243 
7.000  0.200  1.000  0.25726 
laadadax):  3.  4357  C.I.:  0,21794  C.R.:  0.37559 


ftaintainaDilitv 

1  2  3  4  Priori  tv 

1.000  0.111  0.200  0.333  0.05865 

9.000  1.000  5.000  0.333  0.34204 

5.000  0.200  1.000  0.200  0.12103 

3.000  3.000  5.000  1.000  0.47827 

laadadax):  4.7787  C.I.:  0.25956  C.R.:  0.28840 


Haneuverabilitv 
1  2  3 

1.000  ’,000  9.000 
0.143  1.000  '.000 
;.lll  0.  143  1.000 
laadaiaa,.) :  3.3272 


Priority 

0.'6076 

0.19119 

0.04805 


C.P,.:  0.28245 


The  Consistency  a*  ttiis  nierarcfiy  =  0.2®9? 
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APPENDIX  J 


AHP  Follow-Up  Survey  Data 


Judge  t  4  AHP  RESULTS 
TAV  ETfectiveness 


1 

•> 

3 

4 

5 

6 

Priority 

1.000 

9.000 

9.000 

9.000 

9.000 

9.000 

0.60031 

0.111 

1.000 

5.000 

1.000 

3.000 

3.000 

0.12656 

0.111 

0.200 

1.000 

1.000 

0.333 

0.333 

0.03987 

0.111 

1.000 

1.000 

1.000 

3.000 

3.000 

0.10375 

0.111 

0.333 

3.000 

0.333 

1.000 

0.200 

0.04667 

0.111 

0.333 

3.000 

0. 333 

5.000 

1.000 

0.08285 

laKja(aax):  &.9923 

C.I.: 

0.19846  C.R 

.:  0.16004 

1 

Reouireients 

1  T 

4 

e 

6 

> 

a 

Priority 

1.000 

9.m 

9.000 

1.000 

1.000 

9.000 

9.000 

9.000 

0.25351 

0.111 

1.000 

1.000 

0.2')0 

1.000 

0.143 

7.000 

0.200 

0.03998 

0.111 

1.000 

1.000 

0.143 

0.143 

0.143 

5.000 

1.000 

0.02849 

1.000 

5.000 

7.000 

1.000 

7.000 

9.000 

9.000 

9.000 

0.33175 

1.000 

1.000 

7.000 

0.143 

1.000 

’.000 

9.000 

9.000 

0.18296 

0.111 

’.000 

7.000 

O.lll 

0. 143 

1.000 

9.000 

0.143 

0.06652 

0.111 

0.143 

0.200 

0.11! 

O.lll 

0.111 

1.000 

0.111 

0.01108 

0.111 

5.000 

1.000 

0.111 

0.111 

7.000 

9.000 

1.000 

0.08570 

laadaleax):  1 

1.258? 

C.I.: 

0.46555  C.R.:  0.33018 

^erlor nance 

1  2  3 

Priority 

1.000  ’.000  0.  143 

0.19119 

0.143  1.000  0.111 

0.04805 

7.000  ®.000  1.000 

0.76076 

laivdaliax) :  3.3276 

C.I.:  0.16382  C.R.:  0.28245 

1  ^ 

PlsiS 

J 

Priority 

1.000 

5. 000 

’.000 

0.72193 

0.200  I. 000 

7.000 

0.22706 

j.lll 

;.i4: 

1.000 

0.05101 

andaiiax!:  3 

,2085 

C.I.:  0.10423  C.R.:  0.17971 

Lcgi 

.  T 

atics 

Pricntr 

l.OCO  0.143 

0. 1 1 1 

■;.04805 

’.000  l.OOC 

0.143 

0.1911“ 

’.OOO  '.COO 

1.000 

O.’bO’S 

aiii(la<ia«):  3. 

3276 

C.I.:  0.16382  C.P.:  0.28245 
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APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Survivability 

1234567  Priority 
1.1)00  7.000  3.000  7.000  0.333  5.000  0.200  0.17865 

0.143  1.000  0.333  3.000  0.200  1.000  0.200  0.04804 

0.333  3.000  1.000  3.000  1.000  3.000  0.200  0.10180 

0.143  0.333  0.333  1.000  0.200  3.000  0.143  0.03947 

3.000  5.000  1.000  5.000  1.000  5.000  0.333  0.19958 

0.200  1.000  0.333  0.333  0.200  1.000  0.143  0.03310 

5.000  5.000  5.000  7.000  3.000  7.000  1.000  0.39935 

law)a(iax):  7.3842  C.I.:  0.  14736  C.R.:  0.11164 


Prooellants 

1  2  3  Priority 

1.000  1.000  0.333  0.20000 
l.OCO  1.000  0.333  0.20000 
3.000  3.000  1.000  0.60000 
Isaflalaa.'if !  3.0000  2.!.:  0.00000  C.P.;  O.COOOO 


lairtainacilit. 

1234  Frioritv 
1.000  0.333  0.200  1.000  0.10685 

3.000  1.000  0.333  l.CiOO  0.20979 

5.000  3.000  1.000  3.000  0.52550 

1.000  1.000  0.333  1.00  0  0.15786 

laada(iiax):  4.1155  C.I.:  0.C3349  C.R.:  0. 04276 


Maneuverabilitv 
1  2  3  frioritv 

1.000  5.000  7.000  0.71471 
0.200  1.000  5.0'00  0.21849 
0.143  O.jC'O  l.OCO  O.O608O 

laffldaiiax):  3.1828  C.I.:  0.09138  C.R.:  0.15’5t 


Consistency  ot  this  Hierarchy  =  0.0919 


APPENDIX  Js  AHP  FolloM~Up  Survey  Data 


Judge  t  3  AHP  RESULTS 
TAV  Effectiveness 


1 

7 

4 

5 

6 

Priority 

1.000 

1.0«30 

1.000 

3.000 

5.000 

7.000 

0.25874 

1.000 

1.000 

1.000 

3.000 

5.000 

7.000 

0.25874 

1.000 

1.000 

1.000 

3.000 

7.000 

7.000 

0.27401 

0.333 

0.333 

0.  wj3 

1.000 

5.000 

7.000 

0.13252 

0.200 

0.200 

0.143 

0.200 

1.000 

3.000 

0.04795 

0.143 

0.143 

0.143 

0.143 

0.333 

1.000 

0.02803 

laflda(iiu):  b. 

,2805 

C.I.: 

0.05609  C.R 

.:  0.04524 

1 

Reauirenents 

•i  y 

J 

4 

e 

j 

6 

/ 

3 

Priority 

1.000 

7.000 

3.000 

5.000 

5.000 

9.000 

0.333 

1.000 

0.2393: 

0.143 

1.000 

3.000 

0.333 

0.200 

5.000 

0.200 

0.200 

0.04415 

o\  3o., 

0.333 

1.000 

A  yyy 

V  1  WW  V 

0. 143 

3.000 

0.200 

0.  jjj 

0.03548 

0.200 

3.000 

3.000 

1.000 

0.200 

5.000 

0.200 

0.333 

0.06154 

0.200 

5.000 

7.000 

5.000 

1,000 

7.000 

0.333 

3.000 

0.17343 

0.111 

0.200 

0.333 

0.200 

0. 143 

1.000 

0.111 

0.111 

0.01610 

3.000 

5.000 

5.000 

5.000 

3.000 

9.000 

l.OOO 

3.000 

0.30151 

1.000 

5.000 

3.000 

3.000 

I'l 

'}*  vvy 

9,000 

0.333 

l.OOO 

0.12847 

lands (nax):  9.2068 

C.I.: 

0.17241  C.R, 

.:  0.1 

Perforiiance 

1 

Priority 

1.000 

5,000  3.000 

0.65864 

0.200 

1.000  1.000 

0.15618 

0.333 

1.  000  1.000 

0.13517 

landalnaxl:  3.0291 

C.I.:  0.01453  C.R.:  0.C25O5 

■  2 

Risks 

Priority 

1.000  0.200 

0.143 

0.07193 

5.000  1.000 

0.333 

0.27895 

7.000  3.000 

1,000 

0,64912 

landa'iiax):  7 

.  0o49 

C.I.:  0.03244  C.R.:  0.05594 

LOClatlCS 

>  V 

^''icrity 

1.000  5,000  0.333 

0.2' 895 

0.20C  1.000  0.143 

o.oop: 

3.000  7.»0  1,000 

0.s49i: 

aiKialnax.):  3.0649 

C.;.:  v..:244  C.R,: 

0.05534 

appendix  Js  AHP  Follow-Up  Survey  Data 


Survivabilitv 

1234547  priority 
1.000  0.333  0.333  0.200  1.000  0.200  0.143  0.03448 

3.000  1.000  1.000  2.000  l.i)00  0.200  0.143  0.07781 

3.000  1.000  1.000  1.000  1.000  0.200  0.143  0.06584 

5.000  0.500  1.000  1.000  3.000  0.333  0.200  0.09141 

1.000  1.000  1.000  0.333  1.000  0.333  0.200  0.05594 

sioOO  5.000  5.000  3.000  3.000  1.000  0.200  0.21440 

7.000  7.000  7.000  5.000  5.000  5.000  1.000  0.45970 

landadax):  7.4524  C.I. !  0.10874  C.R.j  0.082j8 


Prooel  lints 

1  2  3  Priority 

1.000  S.O'X)  0.333  0.28’20 
0.200  1.000  0.147  0,07796 
3.000  4.000  1.000  0.4348* 

laBdi(iix):  3.0940  C.i.:  0.04701  C.R.:  0.O8105 


flaintainatiilitv 

1  2  3  4  Pnaritv 

1.000  l.iWO  0.200  0.200  0.09082 

1.000  1.000  0.333  0.333  0.11701 

5.000  3.000  1.000  1.000  0.39408 

5.000  3.000  1.000  l.OiOO  0.39608 

lamdaliiax):  4.0328  C.I.:  0.01093  C.fi.:  0.01215 


?1aneuverabilitv 
1  2  3  Priority 

1. GOO  0.500  0.200  0,10884 

2, 'jOO  1.000  0.167  0.14258 
5.000  3.0CC  1.000  0.:’2858 

laiiiaa(iiax):  3.0858  C.I.:  0.04288  C.R.:  0.0"?4 

The  Consistency  ot  this  hierarchy  =  0.0472 
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APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Judge  t  2  AHP  RESULTS 


TAV  Effectiveness 

1  2  3 

4 

5 

6 

Priority 

1.000 

1.000 

1.000 

3.000 

1.000 

0.500 

0. 15856 

1.000 

1.000 

2.000 

3.000 

1.000 

1.000 

0.202V6 

1.000 

0.500 

1.000 

1.000 

1.000 

1.000 

0. 14235 

0.333 

0.333 

1.000 

1.000 

0.333 

0.200 

0.06935 

1.000 

1.000 

1.000 

3.000 

1.000 

0.333 

0.15131 

2.000 

1.000 

1.000 

5.000 

3.000 

1.000 

0.27545 

laeda(aax):  6. 

3120 

C.I.: 

0.06240  C.R. 

:  0.05032 

1 

Reauireeents 

n  T 

to  j 

4 

c 

a 

6 

7 

8 

Priority 

1.000 

1.000 

1.000 

0.500 

7.000 

7.000 

2.000 

1.000 

0. 16379 

1.000 

1.000 

1.000 

1.000 

7.000 

7.000 

3.000 

1.000 

0.18690 

1.000 

1.000 

1.000 

1.000 

7.000 

7.000 

2.000 

1.000 

0.17486 

2.000 

1.000 

l.OOO 

1.000 

7.000 

7.000 

1.000 

1.000 

0.18265 

0.143 

0.143 

0.143 

0.143 

1.000 

1.000 

0.333 

0.333 

0.02867 

0.143 

0.143 

0.143 

0.143 

1.000 

1.000 

0.333 

0.333 

0.02867 

0.500 

0.333 

0.500 

1.000 

3.000 

3.000 

1.000 

1.000 

0.09942 

1.000 

1.000 

1.000 

1.000 

3.000 

3.000 

1.000 

1.000 

0.13503 

landaliax):  6. 

,2550 

C.I.: 

0.03643  C.R, 

0.02584 

Perforiiance 


1  2 

j 

Fnarit'. 

1.000  3.000 

1.000 

0.42857 

0.333  l.O'OO 

0.333 

0.14286 

1.000  3.000 

1.000 

0.42857 

lancaieax) :  3, 

.0000 

C.I.:  0.00000  C.R.:  0.00000 

Risks 

1 

Priorit.' 

l.OOO  0. 

1  (‘i 

1 4 J  0.  - J J 

0.08795 

7.000  1, 

, 'WC  3.000 

0.66942 

3.000  0, 

,333  1.000 

0.24264 

andaiiiax: 

3.u07v 

C.I.:  C. 00351  C.R.: 

0.C0605 

.acistics 

*  to  - 

Friarit, 

l.OCO  3.0‘JO  0.333 

j. 25323 

■.333  1.000  0.200 

0.10473 

3.0CO  5.000  1.  000 

0.336*4 

la«clauia>):  3.-325 

0.71*2;:  C.R.:  0.0332C 

APPENDIX  Js  AHP  Follow-Up  Survey  Data 


Survivability 


1 

7 

W 

4 

5 

6 

7 

Prioritv 

1.000 

0.333 

0.333 

0.333 

3.000 

0.200 

0.200 

0.06006 

3.000 

1.000 

1.000 

1.000 

3.000 

1.000 

0.333 

0.14434 

3.000 

1.000 

1.000 

1.000 

3.000 

1.000 

1.000 

0. 16399 

3.000 

1.000 

1.000 

1.000 

3.000 

0.333 

1.000 

0.14434 

0.333 

0.333 

0.333 

0.333 

1.000 

0.333 

0.333 

0.04927 

5.000 

1.000 

1.000 

3.000 

3.000 

1.000 

1.000 

0.21900 

5.000 

3.000 

1.000 

1.000 

3.000 

1.000 

1.000 

0.21900 

laiida(iax):  7 

.4316 

C.I.: 

0.07193 

;  c.R. 

0.05449 

Frooellants 

1  :  3 

Priority 

1.000  0.333 

0.333 

0.14286 

3.000  1.000 

1.000 

0.42857 

3.000  1.000 

1.000 

0.42857 

laida(aax):  3. 

,  0000 

C.I.:  0.00000  C.F;.:  O.iiOOOO 

'liintainabilitv 

1  2  3 

4 

Priority 

1.000  0.333  0.333 

0.333 

0.09393 

3.000  1.000  0.333 

0.500 

0.17614 

3.000  3.000  1.000 

3.000 

0.43268 

3.000  2.000  0.333 

1.000 

0.24725 

laida(iiax) ;  4.2153 

C.I.: 

0.071’5  C.P.:  0.07973 

haneu.erabilit< 

Priority 

l.vOC  0.200 

1 

■Jt  vJ  J 

0.10065 

5.000  1.000 

4,000 

0.67381 

3.000  0.  250 

1.000 

0.22554 

lamaa'iax/:  3< 

r 

C.I.:  0.04238  C.R.:  O.OT^i 

’’he  Coisistencf  3t  this  hierarchy  = 


APPENDIX  J:  AHP  Follow— Up  Survey  Data 


Judge  I  I  AHP  RESULTS 
TAV  EHectiveness 


1 

a 

7 

4 

5 

6 

Priority 

1.000 

3.000 

3.000 

5.000 

1.000 

f 

w* 

,000 

0.31320 

0,333 

1.000 

1.000 

3.000 

0.333 

y 

,000 

0,14129 

0.333 

1.000 

1.000 

3.000 

1.000 

7 

,000 

0.16696 

0."00 

0.333 

•J,  .V. 

1. 000 

0.333 

0 

0.05134 

1.000 

3.000 

1.000 

3.000 

1.000 

T 

,000 

0.:444C 

0.333 

0.333 

Oi 

3.000 

0.333 

1 

000 

0.08280 

landalAix):  b 

.3457 

C.I.: 

0.0691 

■J 

C.R, 

.:  0.05575 

1 

Remreaents 

■?  3 

4 

5 

6 

y 

3 

Priority 

1.000 

1.000 

1.000 

1.000 

1.000 

5.000 

1.000 

1.000 

0.13448 

1.000 

1.000 

1.000 

1.000 

1.000 

5.000 

1,000 

l.OOO 

0.13448 

l.OOO 

1.000 

1.000 

1.000 

1.000 

5.000 

3.000 

1.000 

0. 15496 

1,000 

1 . 000 

1.000 

1 . 000 

1.000 

5.  MO 

3.000 

1.000 

0.  15496 

l.OOO 

1.000 

1.000 

1.000 

l.OOO 

5.0C0 

3.000 

1.000 

0. 15496 

0.200 

0.200 

0.200 

0.200 

0.200 

1.000 

0.200 

0.200 

0.02690 

1.000 

l.OOO 

0.333 

A  7**? 

V  * 

0.333 

5.000 

1.000 

0.333 

0.0843! 

1.000 

1.000 

l.OOO 

1.000 

1.000 

5.000 

3.000 

1.000 

0,15496 

laitdaiiiax!:  B. 

C.I.: 

0.0337 

0  C.R. 

0.0: 

2390 

‘erforeance 

1  2  3 

'Tioritv 

1.000  1.000  3.000 

0.42357 

1,000  1,000  3.000 

0.4235’ 

0.333  0.333  1,000 

0.14286 

lairidaiiiaj;) :  3.0000 

C.I.:  0.00000  C.R,:  0.0000' 

1  ; 

Risks 

'rioritT 

1  .000  0.  200 

0.200 

'O.OiROOl 

5.000  l.WO 

l.OOO' 

0.45455 

5.000  l.'XO 

1.000 

0.45455 

ailda  uTia.n : 

.  0  joo 

...I.:  '.'..'OOiA  C.F.: 

0.0000 

-sq; 

3‘;:5 

1  2 

1.000  0.333 

0,::: 

0.14236 

3.000  1.000 

1.000 

0.4235’ 

:.')oo  l.OOO 

1.000 

0.42857 

aiTidalaa;! ' :  3, 

,  0000 

C.I,:  0.00000  C.F..: 

0.0000 
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APPENDIX  Js  AHP  Follow-Up  Survey  Data 


Judge  t  8  AHP  RESULTS 

TAV  Effectiveness 

1  2  3  4  5 

fa 

Priority 

1.000 

1.000 

1.000 

7.000 

1.000 

7.000 

0.24097 

1.000 

1.000 

1.000 

5.000 

3.000 

5.000 

0.26909 

1.000 

1.000 

1.000 

S.OOO 

1.000 

5.000 

0.21213 

0.143 

0.200 

0.200 

1.000 

0..j0 

0.200 

0.03232 

1.000 

0.333 

1.000 

5.000 

1.000 

5.000 

0. 18442 

0.1*3 

0.200 

0.200 

5.000 

0.200 

1.000 

0.06057 

laida(iax):  fa. 4751 

C.I.: 

0.0950 

2  C.R 

. :  0.07663 

1 

Reaui resents 

'i  ■? 

4 

5 

fa 

7 

8 

Priority 

1.000 

0.200 

9.000 

1.000 

0.333 

9,000 

0.200 

0.200 

0.07520 

5.000 

1.000 

9.000 

3.000 

1.000 

■5.000 

3.000 

0.333 

0,207fa9 

0.111 

0.111 

1.000 

0.143 

0. 143 

1.000 

0. 143 

0,143 

0.01785 

l.OOO 

0 .  v33 

7.000 

1.000 

0.333 

9.000 

3.000 

1.000 

0.12949 

3.000 

1.000 

7.000 

3.000 

1.000 

9.  OOC 

3.000 

1.000 

0,20471 

O.lll 

0.111 

1.000 

0.111 

0.111 

1.000 

0,111 

0.111 

0.01547 

5.000 

0.333 

7.000 

0.333 

0.333 

9.000 

1.000 

0.333 

O.llllfa 

5.0C0 

3.000 

’.000 

1.000 

l.OCO 

9.000 

3.000 

1.000 

0.23842 

laiidadaxl!  ' 

9.0988 

C.I.: 

('.15697  C.R, 

.:  0.11133 

^srf  or  .nance 

Priority 

1.000  5.000  5.000 

0.S8542 

0.200  1.000  5.000 

0.23441 

O.IOO  0.200  1.000 

0.08017 

lamaa(iax):  3.2948 

C.I.:  0.14735  C.R.:  0.25412 

a  n 

Risks 

Priority 

1.000  0.  143 

1.000 

0.13151 

7,000  1.000 

3.000 

0.69406 

;.;C0  0.333 

l.OCO 

0.;’*43 

lasdadax);  3 

.0803 

C.I.:  0.04015  C.R.: 

0.04522 

Logistics 

1 

Priorit* 

I.IOO 

1.000 

o.coo 

0.  13356 

1.000 

1.000 

0.143 

0.11939 

5,000 

7. 000 

l.OCO 

0.74705 

1  diBGd '.ltdX  '  •  VI 

C.I.:  l.LviTO  2.P.: 

0.01)8; 

APPENDIX  Js  AHP  FoIIom— Up  Survey  Data 


Survivability 

1  2  3  4  5  6  7  Priority 

1.000  0.143  3.000  0.143  0.333  1.000  0.143  0.03863 

7.000  1.000  7.000  1.000  7.000  5.000  5.000  0.36441 

0.333  0.143  1.000  0.143  0.143  0.143  0.143  0.02152 

7.000  1.000  7.000  1.000  3.000  7.000  1.000  0.23819 

3.000  0.143  7.000  0.333  1.000  1.000  0.200  0  07296 

1.000  0.200  7.000  0.143  1,000  1.000  0.200  0.05997 

7.000  0.200  7.000  1.000  5.000  5.000  1.000  0.20431 

landa(iax):  7.3369  C.I.:  0.13948  C.ft.:  0.10567 


Prooellants 

1  2  3  Priority 

1.000  9.000  1.000  0.59052 
0.111  1.000  7.000  0.26108 
1.000  0.143  1.000  0.14841 

laidaliias):  5.2304  C.L:  1.11519  C.R.:  1.92274 


Maintainaoilitv 

1234  Priority 
1.000  0.143  0.200  0.200  0.04719 

\0C0  1.000  3.000  5.000  0.53630 

5.000  0.333  l.OOiO  5.000  0.29768 

5.000  0.200  0.200  1.000  0.11883 

laadalaax):  4.3890  C.I.:  0.12968  C.R.:  0.14409 


.’^aneuyerabilitv 
1  2  3  Priority 

1.000  5.000  '.000  0.70149 
0.200  1.000  ’.000  0.23991 
0.143  0.143  1.000  0.05860 
laadaiaax):  3.2948  C.I.:  0.14739  C.R.:  0.25412 

The  Consistency  :f  this  nierarchv  =  0,15'’4 


APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Judge  t  9  m  RESULTS 
TAV  Effectiveness 

123456  Priority 
1.000  0.143  0.333  0.200  0.333  0.333  0.03973 

7.000  1.000  3.000  3.000  1.000  3.000  0.32302 

3.000  0.333  1.000  0.333  0.333  0.333  0.07399 

5.000  0.333  3.000  1.000  3.000  4.000  0.26985 

3.000  1.000  3.000  0.333  1.000  3.000  0.18997 

3.000  0.333  3.000  0.250  0.333  1.000  0.10344 

lanKla(iax):  6.6031  C.I.:  0.  12063  C.R.:  0.09728 


Requireients 

1  2  3  4  5  6  7  3  Priority 

1.000  5.000  7.000  3.000  3.000  7.000  3.000  0.333  0.24621 

0.200  1.000  7.000  1.000  3.000  7.000  3.000  0.333  0.13203 

0.143  0.143  1.000  0.143  0.200  0.333  0.200  0.143  0.01926 

0.333  1.000  7.000  1.000  3.000  5.000  3.000  1.000  0.15014 

0.333  0.333  5.000  0.333  1.000  3.000  3.000  0.200  0.07608 

0.143  0.143  3.000  0.200  0.333  1.000  0.200  0.200  0.02964 

0.333  0.333  5.000  0.333  0.333  5.000  l.OOO  0.200  0.06300 

3.000  3.000  7.000  1.000  5.000  5.000  5.000  1.000  0.28364 

laodaieaxl:  8.9796  C.L:  0.13994  C.R.:  0.09925 


Perfornance 

1  2  3  Priority 

1.000  3.000  0.143  0.15737 
0.333  1.000  0.143  0.07590 
^000  7.000  1.000  0.76623 
laradadax):  3.1356  C.I.:  0.06781  C.R.:  0.11691 


Risks 

1  2  3  Priority 

1.000  1.  000  O.SOO  0.30769 
1.000  1.000  0.300  0.30765 
1.250  1.250  l.OCO  0.38462 
lanaadax):  3.0000  C.I.:  0.00000  C.P.:  0.00000 


Logistics 

1  2  3  Priarit» 

l.OOO  3.000  3.000  O.fiOOOO 
0.333  1.000  1.000  0.20000 
';.333  1.000  1.000  0.20C00 

lasda-iXa:; ) :  3.0000  C.I.:  0.00000  C.R.:  0.00000 
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APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Judge  t  10  AW  RESULTS 
TAV  Effectiveness 

12  3  4  5  6  Priority 

1.000  3.000  3.000  3.000  3.000  3.000  0.34463 

0.333  1.000  3.000  1.000  0.333  0.333  0.09981 

0.333  0.333  1.000  3.000  0.333  0.333  0.08445 

0.333  1.000  0.333  1.000  0.333  0.333  0.06622 

0.333  3.000  3.000  3.000  1.000  0.333  0.16582 

0.333  3.000  3.000  3.000  3.000  1.000  0.23906 

laeda(iax):  6.7049  C.I.:  0.14098  C.R.:  0.11370 


Reaui resents 


1 

J 

4 

5 

6 

S 

Priority 

1.000 

1.000 

3.000 

1.000 

3.000 

3.000 

3.000 

1.000 

0.17917 

1 .  ooo 

1.30C 

1 . 000 

0.333 

1. 000 

3.300 

l.OOC 

1 .  COO 

0.13828 

0.333 

1.000 

1.000 

V.  333 

*  ’'.''A 

3. 300 

1' 

.  •  vwv 

0.0737? 

l.OCC 

3.000 

3.000 

1 .  i'OO 

5.000 

5.000 

3.000 

l.COC 

0.23431 

0.333 

1.000 

1 . 000 

C.200 

i 

1.000 

3.000 

0.333 

c.osri 

0.333 

0.333 

0.333 

0.200 

1,000 

1.000 

1.000 

0.333 

0.05051 

0.333 

1.000 

3.000 

0.333 

0.333 

1.000 

1.000 

0.200 

0.07566 

1.000 

l.OOC 

3.000 

1.000 

3.000 

3.000 

5.000 

1,000 

0.19652 

landalaaxl:  8. 

7258 

C.I.: 

C.1C368  C.R, 

.:  0.07353 

f'er»or«ance 

1  2  3  Prion  t.' 

l.OCO  7.333  0.200  0.09717 
3.000  1.000  0.200  U. 20212 
5.000  5.  00C  1.000  O.^’OOTl 

laadadaxj;  O.lTSs  C.I.:  0.06781  C.R.:  0.11691 


Risks 

1  2  3  Priority 

1.000  0.500  0.500  0.20000 
2.000  1.000  1.000  0.40000 
2.000  1.000  1.000  0.10000 

landa'.ia"! :  3.0000  3. 1.:  0.00000  C.R.:  0.00000 


Ljgistics 

1  2  3  Priorit. 

1.000  3.  OOO  0.111  0.12^52 
0.333  1.000  0.111  C.0622’ 

=  .000  9.000  1.000  0.8C'322 

Isnidi'ia.- ; :  3.  1356  2. 1.:  0.0e33l  C.F,.:  0.1169! 
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APPENDIX  J:  AHP  FolloM-Up  Survey  Data 


Survivability 


1 

U 

3 

4 

5 

6 

l.OOO 

1.000 

0.200 

1.000 

0.333 

0.200 

1.000 

1.000 

0.200 

0.333 

0.333 

0.333 

5.000 

5.000 

1.000 

5.000 

5.000 

5.000 

1.000 

3. 000 

0.200 

1.000 

5.000 

3.000 

J.  000 

3.000 

0.200 

0.200 

1.000 

1.000 

5.000 

3.000 

0.200 

0.333 

1.000 

1.000 

5.000 

3.000 

0.200 

0.333 

0.333 

0.333 

laiida(iax):  8. 

3272 

C.I.: 

0.22121 

C.R. 

ProoBllants 

1 

»> 

j 

Priority 

1.000 

7.000 

3.000 

0.6491, 

1 

0.143 

1.000 

0.200 

0.07193 

0.333 

5.000 

1.000 

0.27095 

laadadax;:  3 

0649 

C.I.i 

0,032  44 

C.R. 

namtamat 

ilitv 

1 

n 

J 

4 

Priority 

1.000 

7.000 

7.000 

5.000 

0.64484 

;'.i43 

1.000 

5.000 

3,000 

0,1®210 

0.143 

0.200 

1.000 

0.200 

0.04425 

O.IOO 

0.333 

5.000 

l.'OOO 

0.11881 

laada(iax):  4,4868 

C.I.: 

0.  16228 

C.R. 

'!a 

neuverability 

a 

Priority 

l.)00 

0. 200 

-'T-r 

*.  t  .‘W  J 

0.11397 

5.000 

l.OCXO 

1.000 

0.48064 

3.000 

1.000 

1.000 

0.40539 

laacaiia:. ::  3 

02'’! 

r  ^  • 

0.0145: 

C.R. 

’’he  3:n5i5tenc 

j‘  th 

i;  "iisrarcbY  = 

0.  l'05O 

7  Priority 
0.200  0.05306 
0.333  0.04107 
5.000  0.40561 
3.000  0.19065 
3.000  0.10537 
3.000  0.12116 
1.000  0.08308 
:  0.16758 


0.05594 


APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Judge  t  11  AHP  RESULFS 
TAV  EHKtiveness 

1  :  3  4  5  6  Priority 

1.000  2.000  7.000  7.000  7.000  7.000  0.39494 

0.500  1.000  7.000  7.000  7.000  7.000  0.32603 

0.143  0.143  1.000  0.200  0.143  0.143  0.02182 

0.143  0.143  5.000  1.000  0.143  5.000  0.07371 

0.143  0.143  7.000  7.000  1.000  2.000  0.13246 

0.143  0.143  7.000  0.200  0.500  1.000  0.05104 

laeda(iix):  7.5967  C.I.:  0.31935  C.R.:  0.25754 


Reaui relents 

1  2  3  4  5  6  7  8  Priority 

1.000  7.000  9.000  7.000  9.000  9.000  9.000  7.000  0.42194 

0.143  1.000  9,000  0.143  9,000  9,000  9.000  5.000  0.14559 

0.111  0.111  1.000  0.143  0.200  1.000  5.000  0.143  0.02372 

0.143  7.000  7.000  1.000  7.000  9.000  7.000  7.000  0,24189 

0.111  0.111  5.000  0.143  1.000  7.000  7,000  0.143  0.04677 

0.111  0.111  l.OOO  O.lll  0.143  1.000  0,143  0.143  0.01344 

0.111  0.111  0.200  0.143  0.143  7.000  l.OOO  0.143  0.02134 

0.143  0.200  7.000  0.143  ’.000  7.000  7.000  l.OOO  0.08532 

laidadax);  11.0889  C.I.:  0.44127  C.R.:  0.31296 


Perfarmance 

1  2  3  Prioritv 

1.000  7.000  7,000  0.75311 
0.143  1.000  5.000  0.18397 
0,143  0.200  1.000  0.06292 

laijdadax):  3. 2948  C.I.:  0.14739  C.R,:  0.25412 


Risks 

1  2  3  Priority 

l.X'O  0.200  0.143  0.06680 
5.000  1.000  0.200  0.21849 
'.C'OO  5.000  1.000  0."1471 

laadadax):  2.1828  C.I.:  O.O^ne  C.R.:  O.lS’St 


Logistics 

1  2  3  Priority 

l.OOC  ’.300  3.000  0.5329'’ 

0.142  1.000  0.143  0.0i2’C 
0.'2:  ’.200  1.000  0.30421 

laadadax):  2.1256  C.I.:  O.Ob’Sl  C.P.:  0.11691 
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APPENDIX  Js  AHP  FolioM-Up  Survey  Data 


1 

Survivability 

2  3 

4 

5 

6 

7 

Priority 

1.000 

0.111 

0.143 

0.143 

0.143 

0.143 

0.143 

0.01580 

9.000 

1.000 

7.000 

5.000 

7.000 

7.000 

7.000 

0.44906 

7.000 

0.143 

1.000 

0.143 

5.000 

5.000 

0.143 

0.07343 

7.000 

0.200 

7.000 

1.000 

7.000 

5.000 

0.200 

0.14879 

7.000 

0.143 

0.200 

0.143 

1.000 

1.000 

0.143 

0.03519 

7.000 

0. 143 

0. 200 

0.200 

1.000 

1.000 

0.143 

\03613 

7.000 

0.143 

7.000 

5.000 

7.000 

7.000 

1. 000 

0.24161 

laida(iaii) :  8.9959 

C.I.: 

0.33265  C.ft, 

0.25201 

ProDBliants 

1 

M  -J 

Priority 

1.000 

9.000  7.000 

0.76076 

0.111 

1.000  0.  143 

0.04805 

0.143 

7.000  1.000 

0.19119 

laaila(iax):  3.3276 

C.I.:  0.16382  C.R.:  0.28245 

riaintainaDilitv 

1 

4  J 

4 

Priority 

1.000 

0.14!  0.111 

0.200 

0.03372 

’.jOO 

1.000  0.143 

5.000 

0.20401 

9.000 

7.000  l.OOO 

9.000 

0.68270 

5.000 

0.200  0.111 

1.000 

0.07957 

andadax):  4.5853 

C.I.: 

0.19511  C.R.:  0.2167'’ 

flaneuvsrabilitv 

1 

Priority 

1,000  ’.000  ’,000 

0.74186 

0.143  1.000  7.000 

0.2e2’3 

0.143  0.143  1,000 

0.05540 

andainax):  3.435’ 

C.I.:  0.21784  C.R.:  0.3755? 

'ns  C;r5i:t5'’c-.  it  tni3  ^!lerarcflY  =  0.282? 


APPENDIX  J:  AHP  FolloM-Up  Survey  Data 


Judge  t  12  AHP  RESULTS 
TAV  Effectiveness 

123456  Priority 
1.000  0.143  1.000  5.000  5.000  5.000  0.21111 

7.000  1.000  1.000  5.000  3.000  3.000  0.38633 

1.000  1.000  1.000  5.000  1.000  5.000  0.18754 

0.200  0.200  0.200  1.000  0.333  3.000  0.05058 

0.200  0.333  1.000  3.000  1.000  7.000  0.12546 

0.200  0.333  0.200  0.333  0.143  1.000  0.03898 

landadex):  7.2422  C.I.:  0.24843  C.R.;  0.20035 


Reauireeents 

1  2  3  4  5  6  7  8  Priority 

1.000  7.000  7.000  5.000  0.143  0.200  0.200  5.000  0.13624 

0.143  1.000  1.000  0.200  0.143  0.200  0.200  0.200  0.01760 

0.143  1.000  1.000  0.200  0.143  0.333  0.143  5.000  0.06870 

0.200  5.000  5.000  1.000  0.143  3.000  0.143  0.200  0.06332 

7.000  7.000  7.000  7.000  1. 000  5.000  5,000  3.000  0.29507 

5.000  5.000  3.000  0.333  0.200  1.000  0.143  0.200  0.08627 

5.000  5.000  7.000  7.000  0.200  7.000  1.000  0.200  0.18520 

0.200  5.000  0.200  5.000  0.333  5.000  5.000  1.000  0.14759 

laidadaxi:  13.9826  C.I.:  0.84036  C.R,:  0.59600 


•’erforsance 

1  2  3  Priority 

1.000  7.000  1,000  0.48692 
0.143  1.000  0,200  0.07782 
1,000  5.000  1.000  0.43526 

laidadax):  3.0126  C.I.:  0.00630  C.R.:  0.0103e 


1  2  3  Priority 

l.jOO  M.lOO  n.  143  0.07782 
5,000  1.000  1.000  0.43526 
'.000  1.000  1,000  0,48692 

ianicladax;:  3.'il26  C.I.:  0.00o30  3,P„:  ^j.i'l'OSe 


:.33;5f.;a 

1  2  3  '-'I'jrit, 

1,000  0.143  0.  143  0.1-5540 
'.OOC  1.000  0.143  0.20273 
'.300  '..'00  1.000  0.'41£6 
laiidadax,':  3.435'  C.I.: 
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APPENDIX  Jz  AHP  FoUaw-Up  Survey  Data 


Survivability 


1 

2 

3 

4 

5 

6 

7 

Priority 

1.000 

3.000 

7.000 

1.000 

3.000 

3.000 

3.000 

0.29514 

0.333 

1.000 

5.000 

7.000 

3.000 

3.000 

3.000 

0.26087 

0.143 

0.200 

1.000 

1.000 

3.000 

3.000 

0.200 

0.06511 

1.000 

0.143 

1.000 

1.000 

3.000 

3.000 

0.200 

0.09341 

0.333 

0.333 

0.333 

0.333 

1.000 

1.000 

0.143 

0.04181 

0.333 

0.333 

0.333 

0.333 

1.000 

1.000 

0.200 

0.04317 

0.333 

0.333 

5.000 

5.000 

7.000 

5.000 

1.000 

0,20049 

lanla(iax):  8.4143 

C.I.: 

0.23572 

C.R. 

1  0.17858 

Frooellants 

1 

Priority 

1.000 

9.000 

5.000 

0.73519 

0.111 

1.000 

0,200 

0.05811 

0.200 

5.000 

1.000 

0.20670 

lands (lax):  3. 

1171 

C.I.: 

0.05855 

C.P. 

:  0.10095 

ftaintainaDilitv 

1 

4 

Priority 

1.000 

5.000 

0.200 

0.200 

0.11883 

■;'.:oo 

1.000 

0.200 

0.143 

0.04719 

5.000 

5.000 

l.COO 

0.333 

0.29768 

5.0CO 

’.000 

3.000 

1.000 

0.53630 

landanax):  4. 

,3890 

C.I.; 

0.12968 

C.R. 

!  0. 144<?9 

.taneuvsrabilitv 

1 

2 

Priority 

1.000 

7.000 

3.000 

0.64912 

0.143 

1 . 000 

0.200 

0.07103 

'j.rrr 

5. 000 

1.000 

0.276°5 

iamdaiiax);  3. 

,0649 

C.I.: 

0.03244 

C.P. 

:  O.i 

:'55'’4 

"'■is  -  jnsistenCf  of  this  hierarchy  =  0.707b 


APPENDIX  J:  AHP  Fallatui— Up  Survey  Data 


3eo«etric  flean  for  9  Returned  Surveys 
TAV  Effectiveness 


7 

J 

4 

5  6 

Meiaht 

1.00 

1.22 

1.87 

3.62 

2.31  1.87 

0.27804 

0.82 

1.00 

2.25 

2.84 

2.25  2.31 

0.26297 

0.53 

0.44 

1.00 

1.09 

1.11  1.00 

0.12311 

0.23 

0.35 

0.92 

1.00 

1.29  0.76 

0. 10227 

0.43 

0.44 

0.90 

0.77 

1.00  0.98 

0.10875 

0.53 

0.43 

1.00 

1.32 

1.02  1.00 

0. 12486 

laiKia(eax):  6.05&3 

C.I.: 

0.01126  C.R 

:  0.00906 

Reauireients 

1 

i 

n 

yj 

4 

5  6 

I”  8 

Meioht 

Co«o-Mt 

1.00 

2.08 

3.91 

2.10 

1.39  7.47 

1.44  1.07 

0.22977 

0.0639 

0.48 

1.00 

2.77 

1.20 

0.34  4.02 

1.68  0.64 

0.13445 

0.0374 

0.26 

0. 36 

1.00 

0.36 

0.44  1.71 

0.77  0.55 

0.06119 

0.0170 

0.48 

0.B4 

2.76 

1.00 

1.02  5.70 

1.60  1.08 

0.14541 

0.0404 

0.53 

1.20 

Z* 

0.98 

1.00  4.81 

!.:’6  1.28 

0.15042 

0.0413 

0.13 

0.25 

0.58 

0.18 

0.21  1.00 

0.41  0.20 

0.03161 

0.0083 

0.70 

0.59 

1.29 

0.62 

0.57  2.44 

1.00  0.59 

0.09281 

0.0258 

0.94 

1.56 

1.32 

0.93 

0.78  4.95 

l.’O  I. 00 

0.15434 

0.042? 

landaliax):  8. 

1489 

C.I.: 

C.  02128  C.R. 

:  0.01509 

Per<or*ance 

1 

•i 

Meight 

Comc-Wt 

1.00 

2.39 

1.39 

0.48940 

0.1287 

0.!5 

1.00 

0.58 

0.18062 

0.04:’5 

0.72 

1.7! 

1 . 00 

0. 32999 

0.0868 

Uiidaifflax):  3.0043 

u  ■  i  *  ■ 

0.00213  C.R. 

:  0.0036B 

^isXs 

1 

Keiqnt 

Caso-Wt 

1.00 

0.73 

:.oo 

0.30091 

O.O'COS 

1.31 

1  •  ?v 

i  • .'  > 

0.40133 

0.0410 

1.00 

0.': 

1.00 

0.29776 

O.ijCOj 

iaedaliiax/ ;  3 

0002 

'«  •  i  •  1 

0.00011  C.R. 

;  0.00019 

.  I  j 

atl  13 

1*51 

3:.M-i4t 

1 . 0']' 

1  .!C 

0.0251 

i.  ,  h"* 

I  •  ' 

V. 16012 

0.'7174 

- 

•  *,  * 

1 . 

O.zO'SSS 

\  ' ' 9sl 

IsMa'iia-  :  3 

M'l!  ' 

r  , 

L  •  4  *  • 

?.  Ci^'. 
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APPENDIX  J:  AHP  FolloM-Up  Survay  Data 


1 

Survivability 

2  3 

4 

5 

6 

i 

Height 

COM-ltt 

1.00 

0.44 

0.46 

0.46 

0,67 

0.44 

0.19 

0.05984 

0.0075 

2.26 

1.00 

0.76 

1.52 

1.40 

0.65 

0.39 

0.12576 

0.0157 

2.18 

1.31 

1.00 

1.34 

1.51 

0.72 

0.47 

0.13918 

0.0174 

2.17 

0.66 

0.75 

1.00 

1.14 

0.81 

0.33 

0.10829 

0.0135 

1.48 

0.71 

0.66 

0.88 

1.00 

0.66 

0.27 

0.09183 

0.0115 

^  7*7 

1.53 

1.38 

1.24 

1.51 

1.00 

0.52 

0.15863 

0.0198 

5.26 

2.58 

2.14 

3.00 

3.76 

1.92 

1.00 

0.31648 

0.0395 

laida(iax):  7.0439 

C.I.: 

0.00732 

C.R. 

:  0.00555 

Prooellants 


1 

L 

Height 

1.00 

1.27 

0.43 

0.26101 

0.79 

1.00 

0.68 

0.25917 

2.31 

1.47 

1.00 

0.47981 

la«)aliax):  3.0529  C.I.:  0.0:W3  C.R.:  0.04557 

Haintainability 

!  2  3  4  Height 

1.00  0.25  0.24  0.35  0.08285 

4.03  l.OO  0.90  1.17  0.31238 

4.19  1.12  1.00  1.76  0.36896 

2.32  0.85  0.57  1.00  0.23581  laedadax):  4.0101  C.I.:  0.00336  C.R.!  0.00373 

Maneuver ability 
:  2  3  Weight 

1.00  1.22  2.44  0.43531 

0.82  1.00  3.05  0.40992 

0.41  0.33  1.00  0.15478 

lawlaliax)!  3.0200  C.I.:  O.OlOOO  C.R.:  0.01724 

’he  Cgnsiatencv  of  this  hierarchy  =  0.0097 
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APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Geofletric  Mean  for  12  Revised  Surveys  (9  Returned, 
TAV  Effectiveness 

123456  Weight 
1.000  1.143  2.063  3.869  2.764  2.359  0.29743 

0.875  1.000  2.367  2.945  2.  160  2.205  0.26146 

0.485  0.423  1.000  1.168  0.840  0.897  0.11267 

0.258  0.340  0.356  1.000  0.358  0.927  0.09372 

0.362  0.463  1.190  1.166  1.000  1.122  0.11956 

0.42  4  0.453  1.  127  1.079  0.391  1.000  0.11517 

laadaiiax) :  6.0298  C.I.:  0.00596  C.R.:  0.(W480 


Resuirenents 


1 

i 

n 

7 

4 

5 

3 

t 

3 

Weight 

CoiD-Wt 

1.000 

2.397 

4.305 

2.344 

1.803 

5.200 

1.509 

1.412 

0.23851 

0.0709 

0.417 

1.000 

2.575 

0.776 

0.393 

3.26? 

1.551 

0.715 

0.12097 

0.0360 

0.232 

0.338 

l.OO'O 

0.31" 

0.405 

1.495 

0.731 

0.566 

0.05822 

0.0173 

0.437 

1.33« 

W  <  4  Ww 

1.000 

1.159 

5.554 

1 . 560 

1.090 

0. 15569 

0.0463 

0.555 

1.120 

3.4'(' 

0.9s3 

1.000 

4.3’1 

1.021 

0. 14781 

0.044C 

0.192 

0.306 

0. 309 

0.130 

0.229 

1 . 000 

0.371 

0.204 

0.03474 

0.0103 

0.S63 

0.a45 

0.e41 

0.444 

3.6°9 

1.000 

0.438 

o.':e9'e 

0.02 9 7 

0.’03 

1 . 39° 

l.'s" 

0.°13 

0. 9"° 

4.391 

2.285 

1.000 

0.15423 

0.0459 

iafida'iias ' :  8, 

,153c 

•  r  , 

V  •  .  •  • ' 

’4  3.F., 

.:  0.0 

155t 

-ertoriance 

l.)00  2."’'^  l.ria 
i./oo  ';.534 
..’oO  1.  11  1.0'Jv 

la^ica 'la- ' :  3.  }057 


Height  'Qio-Wt 
0.23595  0.0225 
■1.40100  0.037’ 

0.35305  0.0336 

c. 00011  c.'.:  i.-jO'Ci^ 


3  3  Heiunt  Ccmc-Wt 

1.13:  :-,3s5  ,.2:::-  l.OSi' 
..;'5  1.  i.ii:3=  -.'ii't 

i.y.:  i..x  33544  ..xto 
.sii:a  ta  :  1.,1X  l.i.:  l.XlTt 


'■’.if  3 

'..525  l.:3C 
I.’:'!  1.000  l.ll'c 

.aida"»3X:  3,;  103 


He; git  Conc-Wt 
O.f'SS*  0.1351 
■l.!S3'l  0.0420 
1.33"”  0.0384 
C.I.:  0.00363  C.R.: 
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APPENDIX  J:  AHP  Follow-Up  Survey  Data 


Survivabilitv 

4  n  ? 

1  i 

1.000  0.496  0.488 

4567  Weight 

CoiO'Wt 

0.476  0.632  0.441  0.235  0.06425 

0.0074 

2.017  1.000  0.959 

1.678  1.516  0.853  0.577  0.14831 

0.0171 

2.050  1.043  1.000 

1.208  1.951  1.125  0.480  0.14879 

0.0171 

2.100  0.596  0.828 

1.000  1.625  1.168  0.367  0.12467 

0.0144 

1.582  0.660  0.513 

0.615  1.000  0.735  0.293  0.08863 

0.0102 

2.266  1.172  0.888 

0.856  1.361  1.000  0.500  0.13593 

0,0157 

4.263  1.732  2.083 

2.722  3.411  1.999  1.000  0.28942 

0.0333 

laida’wit);  7.0608 

C.I.:  0.01013  C.R.:  0.00767 

Prooel lints 

1  4. 

Weight 

1.000  2.025  0.786 

0.36869 

0,494  1.000  0,487 

0.19631 

1.272  2.055  1.000 

0.43500 

lamla(iax):  3.0057 

C.I.:  0.002S5  C.R. :  0.00491 

Maintainabilitv 

1  2  3 

4  Weight 

1.000  0.402  0.293 

0.402  0,10645 

2.485  1.000  0.7B3 

1.20  0  0.28210 

3.417  1.277  1.000 

1.464  0.36252 

2.488  0.S33  0.683 

1,000  0.24893 

laidadax):  4.0028 

C.I.;  0.00094  C.R.:  0.00105 

Maneuveratiihtv 

1  2  2 

Weignt 

l.OOC  1.  407  2.297 

0.45672 

0.711  I. 000  2.374 

0.36773 

;',  435  '.\421  1.000 

A  pCSA 

J  1  1  V 

la.iidaMia  3..'15t 

■7^0  r C  1  5^  sr  "  -♦  *• 

^  T  ,  ^  C  • 

'15  ‘"I  =  0.  )‘v0 

J~2B 


Appendix  K:  Voting  Data 


Survivability 


1 

2 

3 

4 

5 

6 

7 

Total 

tteightf 

0.00 

O.SO 

0.00 

O.SO 

0.30 

O.SO 

0.30 

2.50 

0.119 

o.so 

0.00 

0.00 

1.00 

1.00 

0.50 

O.SO 

3.50 

0.167 

1.00 

1.00 

0.00 

1.00 

1.00 

1.00 

1.00 

6.00 

0.286 

0.50 

0.00 

0.00 

0.00 

0.00 

0.50 

0.50 

1.50 

0.071 

0.50 

0.00 

0.00 

1.00 

0.00 

0.50 

0.00 

2.00 

0.095 

0.50 

0.50 

0.00 

0.50 

0.50 

0.00 

O.SO 

2.50 

0.119 

O.SO 

0.50 

0.00 

0.50 

1.00 

0.30 

0.00 

3.00 

0.143 

2eta(u)  =  0.4643  2eta(c)  =  O.SOOO 


Propellants 

1 

2  3 

Total  Heights 

0.00 

0.00  0.00 

0.00 

0.000 

1.00 

0.00  0.50 

1.50 

0.500 

1.00 

O.SO  0.00 

1.50 

0.500 

2eta(u) 

=  0.7500 

2etalc)  =  1.0000 

riaintainability 

1 

2  3 

4 

Total  Heights 

0.00 

0.0)  0.00 

0.00 

0.00  0.000 

1.00 

0.00  0.50 

0.50 

2.00  0.333 

1.00 

0,50  0.00 

0.50 

2.00  0.333 

1. 00 

0.50  0.50 

0.00 

2.00  0.333 

2etaiu) 

s  0.5000 

2eta(c)  =  0.5000 

Maneuverability 

1 

2  3 

Total  Heights 

0.00 

0.5i)  1.00 

1.50 

0.500 

0,50 

0,00  1.00 

1,50 

0.500 

0.00 

0.00  0.00 

0.00 

0.000 

ZetatuJ 

=  0.7500 

Zetalc)  =  l.OOOO 

Appendix  K:  Voting  Data 


Judfi  t  7  Risttlts  (Voting  Itatrix) 

TflV  EffKtivBOfss 


1 

2 

3 

4 

5 

6 

Total 

Heights 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

5.00 

0.333 

0.00 

0.00 

1.00 

1.00 

1.00 

0.50 

3.50 

0.233 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

0.00 

0.00 

1.00 

0.00 

1.00 

0.50 

2.50 

0.167 

0.00 

0.00 

1.00 

0.00 

0.00 

0.00 

1.00 

0.067 

0.00 

0.50 

1.00 

0.50 

1.00 

0.00 

3.00 

0.200 

Zetalul 

*  0.9062 

Zeta(c)  - 

0.9375 

Requireeents 

1 

2 

3 

4 

5 

6 

7 

8  Total  Heights 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00  7.00 

0.250 

0.00 

0.00 

1.00 

0.50 

0.50 

1.00 

1.00 

0.50  4.50 

0.161 

0.00 

0.00 

0.00 

0.00 

0.00 

0.50 

0.50 

0.50  1.50 

0.054 

0.00 

0.50 

1.00 

0.00 

1.00 

1.00 

1.00 

0.50  5.00 

0.179 

0.00 

0.50 

1.00 

0.00 

0.00 

1.00 

1.00 

0.50  4.00 

0.143 

0.00 

0.00 

0,50 

0.00 

0.00 

0.00 

0.50 

0.00  1.00 

0.036 

0.00 

0.00 

0.50 

0.00 

0.00 

0.50 

0.00 

0.00  1.00 

0.036 

0.00 

0.50 

0.50 

0.50 

0.50 

1.00 

1,00 

0.00  4.00 

0.143 

Zetalul 

=  0.7625 

Zetalcl  * 

0.7750 

Pertornwee 

1  2  3  Total  Heights 

0.00  1.00  l.OO  2.00  0.667 

0.00  0.00  0.50  0.50  0.167 

0.00  0.50  0.00  0.50  0.167 

Zetalul  =  0,7500  Zeta(c)  »  1.0000 


Risks 

1  2  3  Total  Heights 

0,00  0.50  0.50  1.00  0.333 

0.50  0.00  0.50  1.00  0.333 

0.50  0.50  0.00  1.00  0.333 

Zetalul  =  0.0000  Zeta(c)  *  0.0000 


Logistics 

1  2  3  Total  Heights 

0.00  1.00  0.50  1.50  0.500 

0.00  U.OO  0.00  0.00  0.000 

0.50  1.00  0.00  1.50  0.500 

Zetalul  *  0.7500  Zeta(c)  =  1.0000 


Appendix  K:  Voting  Data 


Surviviliility 


1 

2  3 

4 

5 

6  7 

Total  Heights 

0.00 

0.00  0.00 

0.50 

0.00 

0.00  0.00 

0.50 

0.024 

1.00 

0.00  0.50 

1.00 

0.50 

0.00  0.00 

3.00 

0.143 

1.00 

0.50  0.00 

0.50 

0.50 

0.00  0.00 

2.50 

O.IH 

0.50 

0.00  0.50 

0.00 

0.50 

0.00  0.00 

1.50 

0.071 

1.00 

0.50  0.50 

0.50 

0.00 

0.50  0.50 

3.50 

0.167 

1.00 

1.00  1.00 

1.00 

0.50 

0.00  0.50 

5.00 

0.238 

1.00 

1.00  1.00 

1.00 

0.50 

0.50  0.00 

5.00 

0.238 

Zetalu) 

=  0.6071 

Zetalc)  = 

0.6429 

Propellants 

1 

2  3 

Total  tieights 

0.00 

1.00  1.00 

2.00 

0.667 

0.00 

0.00  0.00 

0.00 

0.000 

0.00 

1.00  0.00 

1.00 

0.333 

Zetalu) 

=  1.0000 

Zetalc)  = 

1.0000 

Haintainability 

1 

2  3 

4 

Total  Heights 

0.00 

0.00  0.00 

0.50 

0.50 

0.083 

1.00 

0.00  0.00 

1.00 

2.00 

0.333 

1.00 

1.00  0.00 

1.00 

3.00 

0.500 

0.50 

0.00  0.00 

0.00 

0.50 

0.083 

Zetalu) 

s  0.8750 

Zetalc)  = 

1.0000 

Haneuverability 

1 

2  3 

Total  Heights 

0.00 

1.00  1.00 

2.00 

0.667 

0.00 

0.00  0.50 

0.50 

0.167 

0.00 

0.50  0.00 

0.50 

0.167 

Zetalu) 

=  0.7500 

Zetalc)  = 

1.0000 

Appendix  K:  Voting  Data 


Judge  t  6  Results  (Voting  Itatrix) 
TAV  Effectiveness 


1 

2 

3 

4  5 

6 

Total  Heights 

0.00 

0.00 

1.00 

1.00  1.00 

0.00 

3.00 

0.200 

1.00 

0.00 

1.00 

1.00  1.00 

0.50 

4.50 

0.300 

0.00 

0.00 

0.00 

0.00  1.00 

0.00 

1.00 

0.047 

0.00 

0.00 

1.00 

0.00  0.50 

0.00 

1.50 

0.100 

0.00 

0.00 

0.00 

0.50  0.00 

0.00 

0.50 

0.033 

1.00 

0.50 

1.00 

1.00  1.00 

0.00 

4.50 

0.300 

Zeta(u) 

=  0.8750 

Zeta(c)  = 

0.9375 

Requi resents 

1 

2 

3 

4  5 

6 

7 

3  Total  Heights 

0.00 

1.00 

1.00 

1.00  1.00 

1.00 

0.00 

0.50  5.50 

0.196 

0.00 

0.00 

1.00 

1.00  0.00 

1.00 

0.00 

0.50  3.50 

0.125 

0.00 

0.00 

0.00 

0.00  0.50 

1.00 

0.00 

0.00  1.50 

0.054 

0.00 

0.00 

1.00 

0.00  0.00 

1.00 

0.00 

0.00  2.00 

0.071 

0.00 

1.00 

0.50 

1.00  0.00 

1.00 

0.00 

0.50  4.00 

0.143 

0.00 

0,00 

0,00 

0.00  0.00 

0.00 

0.00 

0.00  0.00 

0.000 

1.00 

1.00 

1.00 

1.00  1.00 

1.00 

0.00 

1.00  7,00 

0.250 

0.50 

0.50 

1.00 

1.00  0.50 

1.00 

0.00 

0.00  4,50 

0.161 

Zeta(u) 

»  0.8500 

Zeta(c)  = 

0.8750 

Perforiance 

1 

4 

3 

Total  Heights 

0.00 

0.50 

1.00 

1.50  0.500 

0.50 

0.00 

0.00 

0.50  0.167 

0.00 

1.00 

0.00 

1.00  0.331 

Zetalul 

=  0.2500 

Zetalc)  - 

0.5000 

Risks 

1 

7 

J 

Total  Heights 

0.00 

1.00 

1.00 

2.00  0.467 

0,00 

0.00 

0.50 

0.50  0.167 

0.00 

0.50 

0.00 

0.50  0.147 

Zeti(u) 

=  0. 

7500 

Zetalc)  » 

1.0000 

Logistics 

1 

n 

7 

J 

Total  Heights 

0.00 

1.00 

0.00 

1.00  0.333 

0.00 

0.00 

0.00 

0.00  0.000 

1.00 

1.00 

0.00 

2.00  0.447 

Zeta(u) 

=  1.0000 

Zetalc)  : 

1.0000 

Appendix  Ks  Voting  Data 


Survivibility 

1  2 

3 

4 

S 

6 

7 

Total  Neights 

0.00 

0.00 

1.00 

0.00 

0.30 

O.SO 

0.00 

2.00 

0.095 

1.00 

0.00 

1.00 

1.00 

0.50 

1.00 

0.50 

5.00 

0.238 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

0.00 

1.00 

0.00 

1.00 

1.00 

0.00 

4.00 

0.190 

O.SO 

O.SO 

1.00 

0.00 

0.00 

1.00 

0.00 

3.00 

0.143 

0.50 

0.00 

1.00 

0.00 

0.00 

0.00 

0.00 

1.50 

0.071 

1.00 

O.SO 

1.00 

1.00 

1.00 

1.00 

0.00 

5.  SO 

0.262 

Zeta(u) 

*  0.8393 

Zeta(c)  s 

0.8571 

Propellants 

1 

2  3 

Total  Weights 

0.00 

1.00  0.00 

1.00 

0.333 

0.00 

0.00  0.00 

0.00 

0.000 

1.00 

1.00  0.00 

2.00 

0.667 

Zeta(u) 

=  1.0000 

Zeta(c)  =  1.0000 

Haintainabilitv 

1 

n  y 

^  w 

4 

Total  Weights 

O.CiO 

0.00  0.00 

0.00 

0.00  0.000 

1.00 

0.00  1.00 

0.50 

2.50  0.417 

1.00 

0.00  0.00 

1.00 

2.00  0.333 

l.OO 

0.50  0.00 

0.00 

1.50  0.250 

Zetalul 

*  0.6250 

Zeta(c)  s  0.7500 

Naneuverabilitv 

1 

n  y 

ii  V 

Total  Weights 

0.00 

0.00  0.50 

0.50 

0.167 

1.00 

0.00  1.00 

2.00 

0.667 

0.50 

0.00  0.00 

0.50 

0.167 

Zeta(u) 

=  0.7500 

Zetalcl  =  1.0000 

K-10 


Appendix  K:  Voting  Data 


Judgt  t  5  Results  (Voting  Matrix) 

TAV  Effectiveness 

123456  Total  Heights 
0.00  0.00  0.00  1.00  1.00  0.50  2.S0  0.167 

1.00  0.00  0.50  1.00  1.00  1.00  4.50  0.300 

1.00  0.50  0.00  1.00  1.00  1.00  4.50  0.300 

0.00  0.00  0.00  0.00  1.00  0.00  1.00  0.067 

0.00  0.00  0.00  0.00  0.00  1.00  1.00  0.067 

0.50  0.00  0.00  1.00  0.00  0.00  1.50  0.100 

Zeta(u)  =  0.7500  2eta(c)  =  0.8125 


Regui resents 

1  2 

3 

4 

5 

6 

7 

8 

Total  Heights 

0.00 

0.00 

1.00 

1.00 

0.00 

1.00 

1.00 

0.50 

4.50 

0.161 

1.00 

0.00 

1.00 

1.00 

0.00 

1.00 

1.00 

1.00 

6.00 

0.214 

0.00 

0.00 

0.00 

1.00 

0.00 

1.00 

1.00 

1.00 

4.00 

0.143 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

1.00 

1.00 

3.00 

0.107 

1.00 

1.00 

1.00 

1.00 

0.00 

1.00 

1.00 

1.00 

7.00 

0.250 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.00 

1.00 

0.036 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

0.50 

0.00 

0.00 

0.00 

0.00 

1.00 

1.00 

0.00 

2.50 

0.089 

Zeta(u) 

»  0.9375 

Zeta(c)  3 

0.9500 

Perforeance 

1  2  3  Total  Heights 

0.00  1.00  0.50  1.50  O.SOO 

0.00  0.00  0.00  0.00  0.000 

0.50  1.00  0.00  l.SO  0.500 

Zeta(u)  =  0.7500  Zeta(c)  =  1.0000 


Risks 

1  2  3  Total  Heights 

0.00  1.00  1.00  2.00  0.667 

0.00  0.00  0.50  0.50  0.167 

0.00  0.50  0.00  0.50  0.167 

Zeta(u)  =  0.7500  Zeta(c)  =  1.0000 


Logistics 

1  2  3  Total  Heights 

0.00  0.00  0.00  0.00  0.000 

1.00  0.00  0.00  1.00  0.333 

1.00  1.00  0.00  2.00  0.667 

Zeta(u)  =  1.0000  Zeta(c)  =  1.0000 


K-9 


Appendix  K:  Voting  Data 


Survivtbilltv 


1 

*> 

A 

3 

4 

s 

b 

7 

Total  Neiglits 

0.00 

o.so 

0.00 

0.00 

0.00 

0.00 

0.00 

O.SO 

0.024 

o.so 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

O.SO 

0.024 

1.00 

1.00 

0.00 

1.00 

1.00 

o.so 

o.so 

s.oo 

0.238 

1.00 

l.OO 

0.00 

0.00 

0.00 

0.00 

0.00 

2.00 

0.095 

1.00 

1.00 

0.00 

1.00 

0.00 

0.00 

0.00 

3.00 

0.143 

1.00 

1.00 

0.50 

1.00 

1.00 

0.00 

0.50 

5.00 

0.238 

1.00 

1.00 

o.so 

1.00 

1.00 

0.50 

0.00 

5.00 

0.238 

Zeta(u)  :  0.9107  Zetalc)  »  0.928b 


Propellants 

1 

2 

3 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

0.00 

1.00 

2.00 

0.667 

1.00 

0.00 

0.00 

1.00 

0.333 

Zeta(u) 

=  1.0000 

Zeta(c)  =  1.0000 

naintainabilitv 

1 

2 

3 

4 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.00  0.000 

1.00 

0.00 

1.00 

0.00 

2.00  0.333 

1.00 

0.00 

0.00 

0.00 

1.00  0.167 

1.00 

1.00 

1.00 

0.00 

3.00  0.500 

Zetalu) 

»  1.0000 

Zeta(c)  =  1.0000 

Naneuverabtlitv 

1 

3 

Total  Heights 

0.00 

1.00 

1.00 

2.00 

0.667 

0.00 

0.00 

1.00 

1.00 

0.333 

0.00 

0.00 

0.00 

0.00 

0.000 

Zetalul 

=  1. 

0000 

Zeta(c)  =  1.0000 

K-a 


Appendix  Ks  Voting  Data 


Jitdfc  I  4  KMulti  (Votin9  tUtrix) 
TAV  Efftctiventss 


1 

2 

3 

4 

5 

6 

Total  Hti ghts 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

5.00 

0.333 

0.00 

0.00 

1.00 

O.SO 

1.00 

1.00 

3.50 

0.233 

0.00 

0.00 

0.00 

0.50 

0.00 

0.00 

0.50 

0.033 

0.00 

0.50 

0.50 

0.00 

1.00 

1.00 

3.00 

0.200 

0.00 

0.00 

1.00 

0.00 

0.00 

0.00 

1.00 

0.087 

0.00 

0.00 

1.00 

0.00 

1.00 

0.00 

2.00 

0.133 

Zeta(u) 

»  0.7813 

Zetalc)  « 

0.8125 

KequirMcnts 

1 

2 

3 

4 

5 

k 

7 

8 

Total  Heights 

0.00 

1.00 

1.00 

O.SO 

0.50 

1.00 

1.00 

1.00 

8.00 

0.214 

0.00 

0.00 

O.SO 

0.00 

O.SO 

0.00 

1.00 

0.00 

2.00 

0.071 

0.00 

0.50 

0.00 

0.00 

0.00 

0.00 

1.00 

0.50 

2.00 

0.071 

0.50 

1.00 

1.00 

0.00 

1.00 

1.00 

1.00 

1.00 

6.50 

0.232 

0.50 

0.50 

1.00 

0.00 

0.00 

1.00 

1.00 

1.00 

5.00 

0.179 

0.00 

1.00 

1.00 

0.00 

0.00 

0.00 

1.00 

0.00 

3.00 

0.107 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

0.00 

1.00 

O.SO 

0.00 

0.00 

1.00 

1.00 

0.00 

3.50 

0.125 

Zeta(u)  *  0.8123  Zeta(c)  »  0.8250 


Perforunce 


1 

2  3 

Total  Heights 

0.00 

1.00  0.00 

1.00 

0.333 

0,00 

0.00  0.00 

0.00 

0.000 

1.00 

1.00  0.00 

2,00 

0.887 

Zetalul 

=  1.0000 

Zetalcl 

»  1.0000 

Risks 

1 

2  3 

Total  Heights 

0.00 

1.00  1.00 

2.00 

0.887 

0.00 

0,00  1,00 

1.00 

0.333 

0.00 

0.00  0,00 

0.00 

0.004 

Zetalul 

=  1.0000 

Zetalcl 

=  1.0000 

Logistics 

1 

2  3 

Total  Heights 

0.00 

0.00  0.00 

0.00 

0.000 

1.00 

0.00  0.00 

1.00 

0.333 

1.00 

1.00  0.00 

2.00 

0.887 

Zetalul 

»  1.0000 

Zetalcl 

=  1.0000 

Appendix  K:  Voting  Data 


Survivability 
1  2 

3 

4 

5 

6 

7 

Total  Heights 

0.00 

1.00 

1.00 

1.00 

0.00 

1.00 

0.00 

4.00 

0.190 

0.00 

0.00 

0.00 

1.00 

0.00 

0.50 

0.00 

1.50 

0.071 

0.00 

1.00 

0.00 

1.00 

O.SO 

1.00 

0.00 

3.50 

0.167 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.00 

1.00 

0.048 

1.00 

1.00 

o.so 

1.00 

0.00 

1.00 

0.00 

4.50 

0.214 

0.00 

0.50 

0.00 

0.00 

0.00 

0.00 

0.00 

O.SO 

0.024 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.00 

6.00 

0.236 

Zeta(u) 

=  0.8929 

Zetalc)  = 

0.9286 

Propellants 

1 

2  3 

Total  Heights 

0.00 

O.SO  0.00 

0.50 

0.167 

O.SO 

0.00  0.00 

O.SO 

0.167 

1.00 

1.00  0.00 

2.00 

0.667 

Zeta(u) 

*  0.7500 

Zetalc)  =  1.0000 

Raintainabilitv 

1 

0  y 

4 

Total  Heights 

0.00 

0.00  0.00 

O.SO 

O.SO  0.083 

1.00 

0.00  0.00 

0.50 

1.50  0.250 

1.00 

1.00  0.00 

1.00 

3.00  0.500 

0.50 

0.50  0.00 

0.00 

1.00  0.167 

Zeta(u) 

=  0.6250 

Zetalc)  =  0.7500 

haneuverabilitY 

1 

2  3 

Total  Heights 

0.00 

1.00  1.00 

2.00 

0.667 

0.00 

0.00  1.00 

1.00 

0.333 

0.00 

0.00  0.00 

0.00 

0.004 

Zetalui 

=  1.0000 

Zeta(c)  :  1.0000 

APPENDIX  K<  Voting  Data 


Judgt  •  3  Resultf  (Voting  Hatrix) 
TAV  EfiectivMlss 


1 

2 

3 

4 

5  6 

Total  Heights 

0.00 

O.SO 

o.so 

1.00 

1.00  1.00 

4.00 

0.267 

o.so 

0.00 

o.so 

1.00 

1.00  1.00 

4.00 

0.267 

o.so 

o.so 

0.00 

:.oo 

1.00  1.00 

4.00 

0.267 

0.00 

0.00 

0.00 

0.00 

1.00  1.00 

2.00 

0.133 

0.00 

0.00 

0.00 

0.00 

0.00  1.00 

1.00 

0.067 

0.00 

0.00 

0.00 

0.00 

0.00  0.00 

0.00 

0.000 

Zetalu) 

::  0.8750 

Zita(c)  «  0.8750 

Reguirenents 

1 

2 

3 

4 

S  6 

7 

8  Total  Heights 

0.00 

1.00 

1.00 

i.vO 

1.00  1.00 

0.00 

O.SO  5. SO 

0.196 

0.00 

0.00 

1.00 

0.00 

0.00  1.00 

0.00 

0.00  2.00 

0.071 

0.00 

0.00 

0.00 

0.00 

0.00  1.00 

0.00 

0.00  1.00 

0.036 

0.00 

1.00 

1.00 

0.00 

0.00  1.00 

0.00 

0.00  3.00 

0.107 

0.00 

1.00 

1.00 

1.00 

0.00  1.00 

0.00 

1.00  5.00 

0.179 

0.00 

0.00 

0.00 

0.00 

0.00  0.00 

0.00 

0.00  0.00 

0.000 

1.00 

1.00 

1.00 

1.00 

1.00  1.00 

0.00 

1.00  7.00 

0.250 

0.50 

1.00 

1.00 

1.00 

0.00  1.00 

0.00 

0.00  4.50 

0.161 

Zeta(u) 

=  0.9625 

Zetalc)  »  0.9750 

Periorsance 

1 

a 

V 

Total  Heights 

0.00 

1.00 

1.00 

2.00 

0.667 

0.00 

0.00 

0.50 

0.50 

0. 167 

0.00 

0.50 

0.00 

O.SO 

0.167 

Zetalul 

*  0.7500 

Zetalc)  -  1.0000 

Risks 

1 

n 

a 

3 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

0.00 

0.00 

1.00 

0.333 

l.OO 

1.00 

0.00 

2.00 

0.667 

Z9ta(u) 

=  1.0000 

Zetalc)  =  1.0000 

Logistics 

1 

2 

3 

Total  Heights 

0.00 

1.00 

0.00 

1.00 

0.333 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

1.00 

0.00 

2.00 

0.667 

Zeta(u)  =  1.0000  Zetalc)  =  1.0000 


K-5 


APPENDIX  Ks  Voting  Data 


Surviv4btlity 

1  2  3  4  S  &  7  Total  Hcightf 

0.00  0.00  0.00  0.00  O.SO  0.00  0.00  O.SO  0.024 

1.00  0.00  O.SO  1.00  O.SO  0.00  0.00  3.00  0.143 

1.00  O.SO  0.00  O.SO  0.30  0.00  0.00  2.50  0.119 

1.00  0.(K)  O.SO  0.00  1.00  0.00  0.00  2.30  0.119 

0.30  O.SO  0.30  0.00  0.00  0.00  0.00  l.SO  0.071 

1.00  1.00  1.00  1.00  1.00  0.00  0.00  S.OO  0.238 

1.00  1.00  1.00  1.00  1.00  1.00  0.00  6.00  0.284 

Zeta(u)  3  0.78S7  2cta(c)  =  0.8214 


Propellants 

1  2  3  Total  Heights 

0.00  1.00  0.00  1.00  0.333 

0.00  0.00  0.00  0.00  0.000 

1.00  1.00  0.00  2.00  0.647 

Zeta(u)  =  1.0000  Zetalc)  =  1.0000 

Naintainabilitv 

1234  Total  Heights 
0.00  0.50  0.00  0.00  O.SO  0.083 

0.50  0.00  0.00  0.00  0.30  0.083 

1.00  1.00  0.00  O.SO  2.30  0.417 

1.00  1.00  0.50  0.00  2.S0  0.417 

Zeta(ul  :  0.7500  Zctilc)  -  1.0000 

haniuverabtUty 

I  2  3  Total  Heights 

0.00  0.00  0.00  0.00  0.000 

1.00  0.00  0.00  1.00  0.333 

1.00  1.00  4.00  2.00  0.667 

Zeta(u)  :  1.0000  Zetalc)  =  1.0000 


appendix  Kt  Voting  Data 


Judgt  •  2  RmuUs  (Voting  Hatrix) 
TAV  Effictiveness 


1 

2 

3 

4 

5 

6 

Total 

Heights 

0.00 

0.50 

0.50 

1.00 

0.50 

0.00 

2.50 

0.147 

0.50 

0.00 

1.00 

1.00 

0.50 

0.50 

3.50 

0.233 

0.50 

0.00 

0.00 

0.50 

0.50 

0.50 

2.00 

0.133 

0.00 

0.00 

0.50 

0.00 

0.00 

0.00 

0.50 

0.033 

0.50 

0.50 

0.50 

1.00 

0.00 

0.00 

2.50 

0.147 

1.00 

0.50 

0.50 

1.00 

1.00 

0.00 

4.00 

0.247 

Zeta(u)  *  0.3750  Zeta(c)  *  0.4375 


Reguireaents 

1 

0 

a 

3 

4 

5  4 

7 

8 

Total  Heights 

0.00 

0.50 

0.50 

0.00 

1.00  1.00 

1.00 

O.SO 

4.50 

0.161 

0.50 

0.00 

0.50 

0.50 

1.00  1.00 

1.00 

0.50 

5.00 

0.179 

0.50 

0.50 

0.00 

0.50 

1.00  1.00 

1.00 

0.50 

5.00 

0.179 

1.00 

0.50 

0.50 

0.00 

1.00  1.00 

O.SO 

0.50 

5.00 

0.179 

0.00 

0.00 

0.00 

0.00 

0.00  0.50 

0.00 

0.00 

0.50 

0.016 

0.00 

0.00 

0.00 

0.00 

0.50  0.00 

0.00 

0.00 

0.50 

0.018 

0.00 

0.00 

0.00 

0.50 

1.00  1.00 

0.00 

0.50 

3.00 

0.107 

0.50 

0.50 

0.50 

0.50 

1.00  1.00 

0.50 

0.00 

4.50 

0.161 

Zeta(u) 

=  0.6250 

Zeta(c)  :  0.6500 

PerToraance 

1 

2 

3 

Total  Heights 

0.00 

1.00 

0.50 

1.50 

0.500 

0.00 

0,00 

0.00 

0.00 

0.000 

0.50 

l.OO 

0.00 

i.SO 

0.504 

ZBta(u) 

=  0. 

7500 

Zeta(c)  =  1.0000 

Risks 

1 

2 

J 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

0,00 

1.00 

2.00 

0.647 

1.00 

0,00 

0.00 

1.00 

0.333 

Zeta(u) 

=  1. 

0000 

Zeta(c)  =  1.0000 

Logistics 

1  2  3  Total  Heights 

0.00  1.00  0.00  1.00  0.333 

0.00  0.00  0.00  0.00  0.000 

1.00  1.00  0.00  2.00  0.647 

Zeta(u)  s  1.0000  Zeta(c)  =  1.0000 


APPENDIX  K:  Voting  Data 


Survivability 


1 

2 

3 

4 

5  6 

7 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

1.00  0.00 

0.00 

1.00 

0.048 

1.00 

0.00 

0.50 

0.50 

1.00  0.50 

0.00 

3.50 

0.167 

1.00 

0.50 

0.00 

0.50 

1.00  0.50 

0.50 

4.00 

0.194 

1.00 

0.50 

0.50 

0.00 

1.00  0.00 

0.50 

3.50 

0.167 

0.00 

0.00 

0.00 

0.00 

0.00  0.00 

0.00 

0.00 

0.004 

1.00 

0.50 

0.50 

1.00 

1.00  0.00 

0.50 

4.50 

0.214 

1.00 

1.00 

0.50 

0.50 

1.00  0.50 

0.00 

4.50 

0.214 

Zeta(u) 

=  0.6780 

Zetalcl  3  0.7143 

Propellants 

1 

2 

3 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

0.00 

0.50 

1.50 

0.500 

1.00 

0.50 

0.00 

1.50 

0.504 

Zeta(u) 

=  0.7500 

Zeta(c)  =  1.0000 

Raintainabilitv 

1 

2 

T 

V 

4 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.00  0.000 

l.OO 

0.00 

0.00 

0.00 

1.00  0.167 

1.00 

1.00 

0.00 

1.00 

3.00  0.500 

1.00 

1.00 

0.00 

0.00 

2.00  0.333 

Zetalul 

-  1.0000 

Zeta(c)  »  1.0000 

rtaneuverabilitv 

1 

7 

J 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

0.00 

1.00 

2.00 

0.667 

1.00 

0.<)0 

0.00 

1.00 

0.333 

Zetaiu) 

=  1.0000 

Zeta(c)  =  1.0000 

K-2 


appendix  K:  Voting  Data 


Judqe  t  1  Results  (Voting  Hatrix) 
TAV  Ellectiveness 


1 

2 

3 

4 

5 

b 

Total 

Heights 

0.00 

1.00 

1.00 

1.00 

0.50 

1.00 

4.50 

0.300 

0.00 

0.00 

0.50 

1.00 

0.00 

1.00 

2.50 

0.147 

0.00 

0.50 

0.00 

1.00 

0.50 

1.00 

3.00 

0.200 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

0.50 

1.00 

0.50 

1.00 

0.00 

1.00 

4.00 

0.267 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.067 

Zeta(u) 

=  0.8437 

Zeta(c)  : 

0.8750 

Regui resents 

1 

2 

3 

4 

5 

b 

7 

8  Total  Heights 

0.00 

0.50 

0.50 

0.50 

0.50 

1.00 

0.50 

0.50  4.00 

0.143 

0.50 

0.00 

0.50 

0.50 

0.50 

1.00 

0.50 

0.50  4.00 

0.143 

0.50 

0.50 

0.00 

0.50 

0.50 

1.00 

1.00 

0.50  4.50 

0.161 

0.50 

0.50 

0.50 

0.00 

0.50 

1.00 

1.00 

0.50  4.50 

0.161 

0.50 

0.50 

0.50 

0.50 

0.00 

1.00 

1.00 

0.50  4.50 

0.161 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00  0.00 

0.000 

0.50 

0.50 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00  2.00 

0.071 

0.50 

0.50 

0.50 

0.50 

0.50 

1.00 

1.00 

0.00  4.50 

0.161 

Zetalul 

=  0.4250 

Zeta(c)  = 

0.4500 

Perforaance 

1  2  3  Total  Heights 

0.00  0.50  l.OO  1.50  0.500 

0.50  0.00  1.00  1.50  0.500 

0.00  0.00  0.00  0.00  0.000 

Zeta(u)  =  0.7500  Zeta(c)  =  1.0000 


Risks 

1  2  3  Total  Heights 

0.00  0.00  0.00  0.00  0.000 

1.00  0.00  0.50  1.50  0.500 

1.00  0.50  0.00  1.50  0.500 

:eta(u)  =  0.7500  Zeta(c)  =  1.0000 


Logistics 

I  2  3  Total  Heights 

0.00  0.00  0.00  0.00  0.000 

1.00  0.00  0.50  1.50  0.500 

1.00  0.50  0.00  1.50  0.500 

Zetalul  =  0.7500  Zetalc)  =  1.0000 


K-1 


Appendix  K:  Voting  Data 


Judge  t  8  Results 
TAV  Effectiveness 


(Voting  Matrix) 


1 

2 

3 

4 

5 

6 

Total  Weights 

0.00 

0.50 

O.SO 

1.00 

0.50 

1.00 

3.50 

0.233 

0.50 

0.00 

0.50 

1.00 

1.00 

1.00 

4.00 

0.267 

O.SO 

0.50 

0.00 

1.00 

0.50 

1.00 

3.50 

0.233 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

0.50 

0.00 

0.50 

1.00 

0.00 

1.00 

3.00 

0.200 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.067 

Zeta(u) 

=  0.7188 

Zeta(c)  = 

0,7500 

Requi resents 

1 

n 

3 

4 

5 

6 

7 

8  Total  Weights 

0.00 

0.00 

1.00 

0.50 

0.00 

1,00 

0.00 

0.00  2.50 

0.089 

1.00 

0.00 

1.00 

1.00 

0.50 

1.00 

1.00 

0,00  5.50 

0.196 

0.00 

0.00 

0.00 

0.00 

0.00 

0.50 

0.00 

0.00  0.50 

0.018 

0.50 

0.00 

1.00 

0.00 

0.00 

1.00 

1.00 

0.50  4.00 

0.143 

1.00 

0.50 

1.00 

1.00 

0,00 

1.00 

1.00 

0.50  6.00 

0.214 

0.00 

0.00 

0.50 

0.00 

0,00 

0.00 

0.00 

0.00  0.50 

0.018 

1.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.00 

0.00  3.00 

0.107 

1.00 

1.00 

1.00 

0.50 

0.50 

1.00 

1.00 

0.00  6.00 

0.214 

2eta(u) 

=  0.8500 

Zeta(cl  = 

0.8750 

Perforeance 

1 

3 

Total  Weights 

0.00 

I.IXI 

1.00 

2,00 

0.667 

0.00 

0.00 

1.00 

1.00 

0.333 

0.00 

0.00 

0.00 

0.00 

0.000 

Zetaiu) 

1  =  1.0000 

2eta(c)  = 

1.0000 

1  2  3  Total  Weights 

0.00  0.00  0.50  0.50  0.  167 

1.00  0.00  l.oO  2.00  0.667 

0.50  0.i>j  0.00  0.50  0.167 

Zetaiu)  =  0.7500  Zetalcl  =  l.OOOO 


Logistics 

1  2  3  Total  Weights 

0.00  0.50  0.00  0.50  CM6'' 

0.50  0.00  0.00  0.50  0.167 

1.00  1.00  C.OO  2.00  O.ooi 


Appendix  Ks  Voting  Data 


Survivitaility 


1 

2 

3 

4 

5 

6 

7 

Total  Heights 

0.00 

0.00 

1.00 

0.00 

0.00 

0.50 

0.00 

1.50 

0.071 

t.oo 

0.00 

1.00 

0.50 

1.00 

1.00 

1.00 

5.50 

0.262 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0,00 

0,00 

0.000 

l.OO 

0.50 

1.00 

0.00 

1.00 

1.00 

0.50 

5.00 

0.23B 

1.00 

0.00 

1.00 

0.00 

0.00 

0.50 

0.00 

2.50 

0.U9 

0.50 

0.00 

1.00 

0.00 

0.50 

0.00 

0.00 

2.00 

0.095 

1.00 

0.00 

1.00 

0.50 

1.00 

1.00 

0.00 

4.50 

0.214 

Zetalul 

=  0.8929 

Zetalc)  = 

0.9286 

Propellants 

1 

n 

L 

Total  Heights 

0.00 

1.00 

0.50 

1.50 

0.500 

0.00 

0.00 

1.00 

1.00 

0.333 

0.50 

0.00 

0,00 

0.50 

0,16? 

Zeta(u) 

=  0. 

2500 

Zeta(c)  = 

0.5000 

daintainahilitv 

1 

0 

3 

4 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.00 

0,000 

1.00 

0.00 

1.00 

1,00 

3.00 

0.500 

1.00 

0.00 

0.00 

1.00 

2.00 

0.333 

1.00 

0.00 

0.00 

0.00 

1.00 

0.167 

Zeta(u) 

s  1.0000 

Zetalc)  = 

1.0000 

fldneuverabilitv 

1  I  j  Tatal  Weights 
0.00  1,00  1.00  2.00  0.667 

0.00  0.00  1.00  1.00  0.333 

0.0(1  0,00  0.00  0.00  0.000 

Zetalul  =  l.OOOO  Zeta(c)  =  1.0000 


Appendix  Ks  Voting  Data 


Judgf  t  9  Results  (Voting  Hatrix) 
TAV  EHectiveness 


1 

0 

A 

3 

4 

5 

4 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

0.00 

1.00 

1.00 

0.50 

1.00 

4.50 

0.300 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.047 

1.00 

0.00 

1.00 

0.00 

1.00 

1.00 

4.00 

0.247 

1.00 

o.so 

1.00 

0.00 

0.00 

1.00 

3.50 

0.233 

1.00 

0.00 

1.00 

0.00 

0.00 

0.00 

2.00 

0.133 

2eta(u) 

=  0.9042 

Zeta(c)  = 

0.9375 

Requireaents 

1 

0 

3 

4 

5 

4 

7 

3  Total  Heights 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.00  4.00 

0.214 

0.00 

0.00 

1.00 

0.50 

1.00 

1.00 

1.00 

0.00  4.50 

0.141 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00  0.00 

0.000 

0.00 

0.50 

1.00 

0.00 

1.00 

1.00 

1.00 

0.50  5.00 

0.179 

0.00 

0.00 

1.00 

0.00 

0.00 

1,00 

1.00 

0.00  3.00 

0.107 

0.00 

0.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00  1.00 

0.034 

0.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.00 

0.00  2.00 

0.071 

1.00 

1.00 

1.00 

0.50 

1.00 

1.00 

1.00 

0.00  4.50 

0.232 

Zeta(u) 

s  0.9375 

Zeta(c)  = 

0.9500 

PerToreance 

1 

2 

3 

Total  Heights 

0.00 

1.00 

0.00 

1.00 

0.333 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

1.00 

0.00 

2.00 

0.447 

Zetalul 

=  1.0000 

Zeta(c)  * 

1.0000 

Risks 

1 

n 

a. 

n 

J 

Total  Heights 

0.00 

0.50 

0.00 

0.50 

0.147 

0.50 

0.00 

0.00 

0.50 

0.147 

1.00 

1.00 

0.00 

2.00 

0.447 

Zeta(u) 

=  0.7500 

Zeta(c)  = 

1.0000 

Looistics 
1  2  3 

0.00  1.00  1.00 
0.00  0.00  0.50 
0.00  0.50  0.00 
Zeti(u)  =  0.7500 


Total  Heights 


2,00 

0.447 

0.50 

0.147 

0,50 

0.167 

Zeta(c) 

=  1.0000 

Appendix  K:  Voting  Data 


Survivability 
1  2 

3 

4 

5 

b 

7 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.048 

1.00 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00 

2.00 

0.095 

1.00 

1.00 

0.00 

0.50 

1.00 

0.50 

0.50 

4.50 

0.214 

1.00 

1.00 

0.50 

0.00 

1.00 

0.50 

0.50 

4.50 

0.214 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

1.00 

0.50 

0.50 

1.00 

0.00 

1.00 

5.00 

0.238 

1.00 

1.00 

0.50 

0.50 

1.00 

0.00 

0.00 

4.00 

0.190 

Zetalu) 

=  0.8393 

Zeta(c)  = 

0.0571 

Propellants 

1 

2  3 

Total  Heights 

0.00 

0.50  0.00 

0.50 

0.1i7 

0.50 

0.00  0.00 

0.50 

0.147 

1.00 

1.00  0.00 

2,00 

0.467 

Zetaiu) 

=  0.7500 

Zeta(c)  =  1.0000 

Naintainability 

1 

2  3 

4 

Total  Heights 

0.00 

0.00  0.00 

0.00 

0.00  0.000 

1.00 

0.00  1.00 

1.00 

3.00  0.500 

1.00 

0.00  0.00 

1.00 

2*00  0«333 

1.00 

0,00  0.00 

0.00 

1.00  0.167 

Zetalu) 

s  1.0000 

Zetalc)  =  1.0000 

naneuverability 

1 

2  3 

Total  Heights 

0.00 

0.00  1.00 

1.00 

0.333 

1.00 

0.00  1.00 

2.00 

0.667 

0,00 

0.00  0.00 

0.00 

0.000 

Zetalu) 

=  1.0000 

Zetalc)  =  1.0000 

Appendix  K:  Voting  Data 


Judge  t  10  Results  (Voting  Matrix) 
TAV  Effectiveness 


1 

2 

3 

4  5 

6 

Total 

Heights 

0.00 

1.00 

1.00 

1.00  1.00 

1.00 

5.00 

0.333 

0.00 

0.00 

1.00 

0.50  0.00 

0.00 

1.50 

0.100 

0.00 

0.00 

0.00 

1.00  0.00 

0.00 

1.00 

0.067 

0.00 

0.50 

0.00 

0.00  0.00 

0.00 

0.50 

0.033 

0.00 

1.00 

1.00 

1.00  0.00 

0.00 

3.00 

0.200 

0.00 

1.00 

1.00 

1.00  1.00 

0.00 

4.00 

0.267 

Zeta(u) 

=  0.9062 

Zeta(c)  = 

0.9375 

Requi resents 

1 

3 

4  5 

6 

7 

8  Total  Weights 

0.00 

0.50 

l.(X) 

0.50  1.00 

1.00 

1.00 

0.50  5.50 

0.196 

0.50 

0.00 

0.50 

0.00  0.50 

1.00 

0.50 

0.50  3.50 

0.125 

0.00 

0.50 

0.00 

0.00  0.50 

1.00 

0.00 

0.00  2.00 

0.071 

0.50 

1.00 

1.00 

0.00  1.00 

1.00 

1.00 

0.50  6.00 

0.214 

0.00 

0.50 

0.50 

0.00  0.00 

0.50 

1.00 

0.00  2.50 

0.089 

0.00 

0.00 

0.00 

0.00  0.50 

0.00 

0.50 

0.00  1.00 

0.036 

0.00 

0.50 

1.00 

0.00  0.00 

0.50 

0.00 

0.00  2.00 

0.071 

0.50 

0.50 

1.00 

0.50  1.00 

1.00 

1.00 

0.00  5.50 

0.196 

Zeta(u) 

:  0.6000 

Zeta(c)  : 

0.6250 

Per foraance 

1 

2 

•7 

w 

Total  Heights 

0.00 

0.00 

0.00 

0.00  0.000 

1.00 

0.00 

0.00 

1.00  0.333 

1.00 

1.00 

0.00 

2.00  0.667 

Zeta(u) 

=  1.0000 

Zeta(c)  = 

1.0000 

Risks 

1 

£ 

3 

Total  Heights 

0.00 

0.00 

0.00 

0.00  0.000 

1.00 

0.00 

0.50 

1.50  0.500 

1.00 

0.50 

0.00 

1.50  0.500 

Zetalu) 

=  0. 

7500 

Zeta(c)  = 

l.'^OOO 

Logistics 

1 

n 

3 

Total  Heights 

0.00 

1.00 

0.00 

1.00  0.333 

0.00 

0.00 

0.00 

0.00  0.000 

1.00 

1.00 

0.00 

2.00  0.6O7 

Zeta(u) 

=  1.0000 

Zetalc)  = 

1.0000 
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Survivability 


1 

2  3 

4 

5  b  7 

Total  Heights 

0.00 

0.50  0.00 

0.50 

0.00  0.00  0.00 

1.00 

0.048 

0.50 

0.00  0.00 

0.00 

0.00  0.00  0.00 

0.50 

0.024 

1.00 

1.00  0.00 

1.00 

1.00  1.00  1.00 

6.00 

0.286 

0.50 

1.00  0.00 

0.00 

1.00  1.00  1.00 

4.50 

0.214 

1.00 

1.00  0.00 

0.00 

0.00  0.50  1.00 

3.50 

0.167 

1.00 

1.00  0.00 

0.00 

0.50  0.00  1.00 

3.50 

0.167 

1.00 

1.00  0.00 

0.00 

0.00  0.00  U.OO 

2.00 

0.095 

Zeta(u) 

0.8214 

Zeta(c)  =  0.8571 

Propellants 

1 

2  3 

Total  Heights 

0.00 

1.00  1.00 

2.00 

0.6i7 

0.00 

0.00  0.00 

0.00 

0.000 

0.00 

1.00  0.00 

1.00 

0.333 

Zeta(u) 

=  1.0000 

Zeta(c)  =  1.0000 

naintainability 

1 

2  3 

4 

Total  Heights 

0.00 

1.00  1.00 

1.00 

3.00  0.500 

0.00 

0.00  1.00 

1.00 

2.00  0.333 

0.00 

0.00  0.00 

0.00 

0.00  0.000 

0.00 

0.00  1.00 

0.00 

l.OO  0.1i7 

Zeta(u) 

s  1.0000 

Zetatc)  =  1.0000 

NaneuverabiUtv 

1 

2  3 

Total  Heights 

0.00 

0.00  0.00 

0.00 

0.000 

1.00 

0.00  0.50 

1.50 

0.500 

1.00 

0.50  0.00 

1.50 

0.500 

Zetalu) 

=  0.7500 

Zeta(c)  1.0000 

T 


T 


1 


Appendix  K:  Voting  Data 


Judge  •  11  Results  (Voting  Hatrix) 
TAV  Effectiveness 


1 

2 

3 

4 

5 

6 

Total  Weights 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

5.00 

0.333 

0.00 

0.00 

1.00 

1.00 

1.00 

1.00 

4.00 

0.267 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

0.00 

0.00 

1.00 

0.00 

0.00 

1.00 

2.00 

0.133 

0.00 

0.00 

1.00 

1.00 

0.00 

1.00 

3.00 

0.200 

0.00 

0.00 

1.00 

0.00 

0.00 

0.00 

1.00 

0,067 

Zeta(u) 

=  1.0000 

Zeta(c)  = 

1.0000 

Requi resents 

1 

*> 

3 

4 

5 

6 

7 

8  Total  Weights 

0.00 

1,00 

1.00 

1.00 

1.00 

1,00 

1.00 

1.00  7.00 

0,250 

0.00 

0.00 

1.00 

0.00 

1.00 

1.00 

1.00 

1.00  5.00 

0.179 

0.00 

0.00 

0.00 

0.00 

0,00 

0,50 

1.00 

0.00  1.50 

0.054 

0.00 

1.00 

1.00 

0.00 

1.00 

1.00 

1.00 

1.00  6.00 

0.214 

0.00 

0,00 

1.00 

0.00 

0.00 

1.00 

1.00 

0.00  3.00 

0.107 

0.00 

0.00 

0.50 

0.00 

0.00 

0.00 

0.00 

0.00  0.50 

0.013 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00  1.00 

0.036 

0.00 

0.00 

1,00 

0.00 

1,00 

1.00 

1.00 

0.00  4.00 

0.143 

Zeta(u) 

s  0.9625 

Zeta(c)  s 

0.9750 

Perforiance 

1  2  3  Total  Weights 

0.00  l.OO  1.00  2.00  0.667 

0.00  0.00  1.00  1.00  0.333 

0.00  0.00  0.00  0.00  0.000 

:eta(u)  =  1.0000  Zeta(c)  =  1.0000 


Risks 

1  2  3  Total  Weights 

0.00  0.00  0.00  0.00  0.000 

1.00  0.00  0.00  1.00  0.333 

1,00  1.00  0.00  2,00  0.667 

Zeta(u)  =  1.0000  Zetalc)  =  1.0000 


Logistics 

1  2  3  Total  Weights 

0.00  1.00  1.00  2.00  0.667 

0.00  0.00  0.00  0.00  0.000 

0.00  1.00  0.00  1.00  0.331 

:eta(u)  =  1.0000  letalc)  =  1.0000 
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Survivability 


1 

2 

3 

4 

5 

b 

7 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

6.00 

0.286 

1.00 

0.00 

0.00 

0.00 

1.00 

1.00 

0.00 

3.00 

0.143 

1.00 

0.00 

1.00 

0.00 

1.00 

1.00 

0.00 

4.00 

0.190 

1.00 

0.00 

0.00 

0.00 

0.00 

0.50 

0.00 

1.50 

0.071 

1.00 

0.00 

0.00 

0.00 

0.50 

0.00 

0.00 

1.50 

0.071 

1.00 

0.00 

1.00 

1.00 

1.00 

1.00 

0.00 

5.00 

0.238 

Zeta(u) 

=  0.9821 

Zetalcl  = 

1.0000 

Propellants 


1 

2  3 

Total  Heights 

0.00 

1.00  1.00 

2.00 

0.667 

0.00 

0.00  0.00 

0.00 

0.000 

0.00 

1.00  0.00 

1.00 

0.333 

Zeta(u) 

=  1.0000 

Zetalcl  =  1.0000 

Haintainability 

1 

0  t 

A  V 

4 

Total  Heights 

0.00 

0.00  0.00 

0.00 

0.00  0.000 

1.00 

0.00  0.00 

1.00 

2.00  0.333 

1.00 

1.00  0.00 

1.00 

3.00  0.500 

1.00 

0.00  0.00 

0.00 

1.00  0.167 

Zeta(u) 

=  l.OOOO 

Zetalcl  »  1.0000 

Maneuverability 

1 

2  3 

Total  Heights 

0.00 

1.00  1.00 

2.00 

0.667 

0.00 

0.00  1.00 

1.00 

0.333 

0.00 

0.00  0.00 

0.00 

0.000 

Zeta(u) 

=  1.0000 

Zetalcl  =  1.0000 
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Judge  I  12  Results  (Voting  Matrix) 
TAV  Eflectiveness 


1 

2 

3 

4 

5 

6 

Total 

Heights 

0.00 

0.00 

O.SO 

1.00 

1.00 

1.00 

3.50 

0.233 

1.00 

0.00 

O.SO 

1.00 

1.00 

1.00 

4.50 

0.300 

o.so 

o.so 

0.00 

1.00 

0.50 

1.00 

3.50 

0.233 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

1.00 

0.067 

0.00 

0.00 

o.so 

1.00 

0.00 

1.00 

2.50 

0.167 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

Zeta(ul 

>  0.8125 

Zeta(c)  » 

0.8750 

Regui resents 

1 

2 

3 

4 

5 

6 

7 

8  Total  Heights 

0.00 

1.00 

1.00 

1.00 

0.00 

0.00 

0.00 

1.00  4.00 

0.143 

0.00 

0.00 

0.50 

0.00 

0.00 

0.00 

0.00 

0.00  0.50 

0.018 

0.00 

0.50 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00  1.50 

0.054 

0.00 

1.00 

1.00 

0.00 

0.00 

1.00 

0.00 

0.00  3.00 

0.107 

1.00 

1.00 

1.00 

1.00 

0.00 

1.00 

1.00 

1.00  7.00 

0.250 

l.OO 

1.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00  3.00 

0.107 

1.00 

1.00 

1.00 

1.00 

0.00 

1.00 

0.00 

0.00  5.00 

0.179 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

1.00 

0.00  4.00 

0.143 

Zeta(ul 

:  0.6625 

Zeta(c)  s 

0.6750 

Perforaance 

1 

a 

3 

Total  Heights 

0.00 

1.00 

o.so 

1.50 

0.500 

0.00 

0.00 

0.00 

0.00 

0.000 

0.50 

1.00 

0.00 

1.50 

0.504 

Zeta(u)  =  0.7500  Zeta(c)  =  t.OOOO 


Risks 

]  2  3  Total  Meights 

0.00  0.00  0.00  0.00  0.000 

1.00  0.00  0.50  1.50  0.500 

1.00  0.50  0.00  1.50  0.500 

Zeta(iJ)  =  0.7500  Zeta(c)  =  1.0000 


Logistics 

1  2  3  Total  Heights 

0.00  0.00  0.00  0.00  0.000 

1.00  0.00  0.00  1.00  0.333 

1.00  1.00  0.00  2.00  0.647 

2eta(u)  =  1.0000  Zetalcl  =  1.0000 
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Survivahility 


1 

4 

3 

4 

5 

6 

7 

Total 

Heights 

0.00 

1.00 

1.00 

o.so 

1.00 

1.00 

1.00 

5.  SO 

0.262 

0.00 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

5.00 

0.238 

0.00 

0.00 

0.00 

o.so 

1.00 

1.00 

0.00 

2.50 

0.119 

0.50 

0.00 

0.50 

0.00 

1.00 

1.00 

0.00 

3.00 

0.143 

0.00 

0.00 

0.00 

0.00 

0.00 

0.50 

0.00 

O.SO 

0.024 

0.00 

0.00 

0.00 

0.00 

0.50 

0.00 

0.00 

0.50 

0.024 

0.00 

0.00 

1.00 

1.00 

1.00 

1.00 

0.00 

4.00 

0.190 

Zetliu) 

=  0. 

8571 

Zetalc)  = 

0.8929 

Propellants 

1 

2  3 

Total  Heights 

0.00 

1.00  1.00 

2.00 

0.447 

0.00 

0.00  0.00 

0.00 

0.000 

0.00 

1.00  0.00 

1.00 

0.333 

Zetalu) 

=  1.0000 

Zetalc)  =  1.0000 

naintainability 

1 

2  3 

4 

Total  Heights 

0.00 

1.00  0.00 

0.00 

1.00  0.147 

0.00 

0.00  0.00 

0.00 

0.00  0.000 

1.00 

1.00  0.00 

0.00 

2.00  0.333 

1.00 

1.00  1.00 

0.00 

3.00  0,500 

Zeta(u) 

=  1.0000 

Zetalc)  =  l.OOM 

Itaneuverabilitv 

1 

2  3 

Total  Heights 

0.00 

1.00  1.00 

2.00 

0.447 

0.00 

0.00  0.00 

0,00 

0.000 

0.00 

1.00  0.00 

1.00 

0.333 

Zetaiu) 

=  1.0000 

Zetalc)  =  1.0000 
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Judge  •  13  Results  (Voting  Matrix) 
TAV  Effectiveness 


1 

2 

3 

4 

5 

6 

Total  Heights 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

s.oo 

0.333 

0.00 

0.00 

1.00 

1.00 

1.00 

1.00 

4.00 

0.267 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.067 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.000 

0.00 

0.00 

1.00 

1.00 

0.00 

1.00 

3.00 

0.200 

0.00 

0.00 

1.00 

1.00 

0.00 

0.00 

2.00 

0.133 

Zeta(u) 

::  1.0000 

Zeta(c)  = 

1.0000 

Regui resents 

1 

2 

3 

4 

5 

6 

7 

8  Total  Heights 

0.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00  7.00 

0.250 

0.00 

0.00 

1.00 

0.00 

0.00 

1.00 

1.00 

0.00  3.00 

0.107 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00  1.00 

0.036 

0.00 

1.00 

1.00 

0.00 

1.00 

1.00 

1.00 

1.00  6.00 

0.214 

0.00 

1.00 

1.00 

0.00 

0.00 

1.00 

1.00 

1.00  5.00 

0.179 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00  0.00 

0.000 

0.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.00 

0.00  2.00 

0.071 

0.00 

1.00 

1.00 

0.00 

0.00 

1.00 

1.00 

0.00  4.00 

0.143 

Zeta(u) 

S  1.0000 

Zeta(c)  = 

1.0000 

Perforeance 


1 

2 

3 

Total  Heights 

0.00 

1.00 

1.00 

2.00  0.667 

0.00 

0.00 

0.00 

0.00  0.000 

0.00 

1.00 

0.00 

1.00  0.333 

Zeta(u) 

s  1.0000 

Zetalc)  =  1.0000 

Risks 


1 

2  3 

Total  Heights 

0.00 

0.00  0.00 

0.00  0.000 

1.00 

0.00  1.00 

2.00  0.667 

1.00 

0.00  0.00 

1.00  0.333 

Zetalul 

=  1.0000 

Zetalc)  :  1.0000 

Logistics 

1 

2  3 

Total  Heights 

0.00 

1.00  0.00 

1.00  0.333 

0.00 

0.00  0.00 

0.00  0.000 

1.00 

1.00  0.00 

2.00  0.667 

Zetalu) 

=  1.0000 

Zetalc)  =  1.0000 
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Survivability 


1 

2 

3 

4 

S  6 

7 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.00  0.00 

0.00 

0.00 

0.000 

1.00 

0.00 

0.00 

1.00 

1.00  0.00 

0.00 

3.00 

0.143 

1.00 

1.00 

0.00 

1.00 

1.00  1.00 

0.00 

5.00 

0.238 

1.00 

0.00 

0.00 

0.00 

1.00  1.00 

0.00 

3.00 

0.143 

1.00 

0.00 

0.00 

0.00 

0.00  0.00 

0.00 

1.00 

0.048 

1.00 

1.00 

0.00 

0.00 

1.00  0.00 

0.00 

3.00 

0.143 

1.00 

1.00 

1.00 

1.00 

1.00  1.00 

0.00 

6.00 

0.286 

Zetalul 

=  0.9286 

Zeta(c)  =  0.9286 

Prooellants 

1 

2 

3 

Total  Heights 

0.00 

1.00 

0.00 

1.00 

0.333 

0.00 

0.00 

0.00 

0.00 

0.000 

1.00 

1.00 

0.00 

2.00 

0.667 

Zetalu) 

=  1. 

0000 

Zetalc)  =  1.0000 

Naintainability 

1 

n 

i 

3 

4 

Total  Heights 

0.00 

0.00 

0.00 

0.00 

0.00  0.000 

1.00 

0.00 

0.00 

1.00 

2.00  0.333 

1.00 

l.OO 

0.00 

1.00 

3.00  0.500 

1.00 

0.00 

0.00 

0.00 

1.00  0.167 

ZetatuI 

’  1.0000 

Zetalc)  s  1.0000 

Naneuverability 

1 

n 

3 

Total  Heights 

0.00 

1.00 

1.00 

2.00 

0.667 

0.00 

0.00 

1.00 

1.00 

0.333 

0.00 

0.00 

0.00 

0.00 

0.000 

Zetalu) 

=  1.0000 

Zetalc)  =  1.0000 
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